[
https://issues.apache.org/jira/browse/CASSANDRA-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532262#comment-14532262
]
Benedict commented on CASSANDRA-9318:
-------------------------------------
I'm very much in favour of stopping C* keeling over, and doing so in 2.1, but I
don't see smooth "rate limiting" as a realistic possibility (in the near-term),
and nor do I think a coordinator managed approach will do the job.
The problem with the coordinator is two-fold:
# It has no idea how big its responses are going to be, so it cannot prevent
new requests from starting that will not, combined, blow its constraints
# It has no idea how many _of its own_ requests are still in flight in the rest
of the cluster
The problem can easily be illustrated by the fact that we can already bring
down clusters trivially with a default stress workload. Stress packets are
tiny, and only ~256 of them can be in-flight at any time with any coordinator.
The combined request size is less than 1Mb, and yet the cluster can be
destroyed. The reason for this is that many more requests are in-flight than
the coordinator realises. The processing nodes need to have size-based load
shedding, as was essentially proposed by CASSANDRA-8518. Or, as a simple
initial mechanism, a bound on the size of the work queue as proposed by
CASSANDRA-5039. This latter approach is probably the easiest to introduce in
2.1.
> Bound the number of in-flight requests at the coordinator
> ---------------------------------------------------------
>
> Key: CASSANDRA-9318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9318
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Ariel Weisberg
> Assignee: Ariel Weisberg
> Fix For: 3.x
>
>
> It's possible to somewhat bound the amount of load accepted into the cluster
> by bounding the number of in-flight requests and request bytes.
> An implementation might do something like track the number of outstanding
> bytes and requests and if it reaches a high watermark disable read on client
> connections until it goes back below some low watermark.
> Need to make sure that disabling read on the client connection won't
> introduce other issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
