[
https://issues.apache.org/jira/browse/CASSANDRA-21571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114304#comment-18114304
]
Rishabh Saraswat commented on CASSANDRA-21571:
----------------------------------------------
I wonder if we could approach this as dynamic admission control rather than
permanently partitioning the executor pool.
For example, suppose a node has 100 units of execution capacity. LWTs should be
able to consume otherwise-unused capacity, so if there is no competing workload
they could use close to all 100. However, once non-LWT work starts becoming
starved(LWT took up all of the capacity), we stop accepting new LWTs beyond
some safe threshold (X% of pool, if there is LWT demand, if not, threshold is
disregarded) and let existing LWT operations drain naturally.
So conceptually:
* No non-LWT demand → LWTs can use the available capacity.
* Non-LWT demand appears → stop admitting additional LWT work once the shared
capacity is under pressure.
* Existing LWTs are not forcibly cancelled; as they complete, their capacity
becomes available to the waiting non-LWT work.
* Once the pressure disappears, LWT admission can increase again.
This avoids permanently reserving, say, 20% of the pool for LWTs, which could
unnecessarily starve LWT throughput when the rest of the pool is idle.
The main question would be what signal should drive admission control (pending
work, queue depth, scheduling latency, etc.) and at which executor/stage this
should be enforced.
> Guardrail to limit LWT per-partition concurrency
> ------------------------------------------------
>
> Key: CASSANDRA-21571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21571
> Project: Apache Cassandra
> Issue Type: Improvement
> Reporter: Elliott Sims
> Priority: Normal
>
> I'm actually not sure if this should be a Guardrail exactly, or something
> else, but:
> Cassandra's threading model combined with LWT behavior means that it's very
> easy for contention on a single partition to tie up all of the threads on the
> hosts that own that partition, which causes cascading failures across the
> cluster and any clients. It's possible to set up rate-limits at the
> application level, but that can be challenging in a distributed system.
> Especially one that involves multiple applications or products.
> So, it would be very nice to have some way to say "only allow N (or X% of a
> pool?) threads to contend LWTs on a partition, and reject any further
> queries" to limit the blast radius to only that partition.
> Even a per-host limit on concurrent LWTs would help a fair amount even if it
> wouldn't completely isolate the problem partition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
