[
https://issues.apache.org/jira/browse/CASSANDRA-21571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Sims updated CASSANDRA-21571:
-------------------------------------
Description:
I'm actually not sure if this should be a Guardrail exactly, or something else,
but:
Cassandra's threading model combined with LWT behavior means that it's very
easy for contention on a single partition to tie up all of the threads on the
hosts that own that partition, which causes cascading failures across the
cluster and any clients. It's possible to set up rate-limits at the
application level, but that can be challenging in a distributed system.
Especially one that involves multiple applications or products.
So, it would be very nice to have some way to say "only allow N (or X% of a
pool?) threads to contend LWTs on a partition, and reject any further queries"
to limit the blast radius to only that partition.
Even a per-host limit on concurrent LWTs would help a fair amount even if it
wouldn't completely isolate the problem partition.
was:
I'm actually not sure if this should be a Guardrail exactly, or something else,
but:
Cassandra's threading model combined with LWT behavior means that it's very
easy for contention on a single partition to tie up all of the threads on the
hosts that own that partition, which causes cascading failures across the
cluster and any clients. It's possible to set up rate-limits at the
application level, but that can be challenging in a distributed system.
Especially one that involves multiple applications or products.
So, it would be very nice to have some way to say "only allow N (or X% of a
pool?) threads to contend LWTs on a partition, and reject any further queries"
to limit the blast radius to only that partition.
> Guardrail to limit LWT per-partition concurrency
> ------------------------------------------------
>
> Key: CASSANDRA-21571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21571
> Project: Apache Cassandra
> Issue Type: Improvement
> Reporter: Elliott Sims
> Priority: Normal
>
> I'm actually not sure if this should be a Guardrail exactly, or something
> else, but:
> Cassandra's threading model combined with LWT behavior means that it's very
> easy for contention on a single partition to tie up all of the threads on the
> hosts that own that partition, which causes cascading failures across the
> cluster and any clients. It's possible to set up rate-limits at the
> application level, but that can be challenging in a distributed system.
> Especially one that involves multiple applications or products.
> So, it would be very nice to have some way to say "only allow N (or X% of a
> pool?) threads to contend LWTs on a partition, and reject any further
> queries" to limit the blast radius to only that partition.
> Even a per-host limit on concurrent LWTs would help a fair amount even if it
> wouldn't completely isolate the problem partition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
