[
https://issues.apache.org/jira/browse/CASSANDRA-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051559#comment-15051559
]
Ariel Weisberg commented on CASSANDRA-9318:
-------------------------------------------
The discussion so far has lead to throwing OE or UE to the client being too
complex starting point since the database will throw off more errors then it
used to.
The advantage of a bound on just memory committed at the coordinator is that
it's "free" from the perspective of the user. OOM and blocking are similar
levels of unavailability. The difference is that OOM is more disruptive. If the
coordinator at least prevents itself from OOMing it can recover without
intervention and continue to serve writes.
I think we are being optimistic about how many writes you can buffer and time
out at coordinators before things start going south. The space amplification is
going to be bad and the GC overhead increases linearly with the number of
timing out writes in flight. It's going to lead to survivor copying and
promotion which is going to lead to fragmentation. I think we will find that we
can't buffer enough writes to tolerate a 2 second timeout in a small cluster
where a single coordinator must coordinate many requests for a slow node. I
think we are going to see what Benedict described, especially in small
clusters, where performance is cyclical as the coordinators block waiting for
timeouts to fire.
I have an implementation of most of this. Fussing over how to calculate the
weight of inflight callbacks and mutations without introducing a lot of
overhead walking the graph repeatedly. We already walk it once to calculate the
serialized size, but that is not the same thing. I usually just go with a rough
guesstimate, serialized size is a good proxy for a weight, but we are trying to
fill the heap reliably. It's bad enough not knowing the layout of the heap or
the garbage collector induced overhead.
> Bound the number of in-flight requests at the coordinator
> ---------------------------------------------------------
>
> Key: CASSANDRA-9318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9318
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Ariel Weisberg
> Assignee: Ariel Weisberg
> Fix For: 2.1.x, 2.2.x
>
>
> It's possible to somewhat bound the amount of load accepted into the cluster
> by bounding the number of in-flight requests and request bytes.
> An implementation might do something like track the number of outstanding
> bytes and requests and if it reaches a high watermark disable read on client
> connections until it goes back below some low watermark.
> Need to make sure that disabling read on the client connection won't
> introduce other issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
