[
https://issues.apache.org/jira/browse/CASSANDRA-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056940#comment-15056940
]
Ariel Weisberg commented on CASSANDRA-9318:
-------------------------------------------
Further review shows that with an 8 gigabyte heap backpressure isn't getting a
chance to kick in. Several memtables end up being queued at once for flushing
but the peak memory utilization only a few hundred megabytes. I am back to not
having a workload that demonstrate that backpressure has value.
I tried artificially constraining disk throughput to 32 megabytes/sec and ended
up hitting OOM in hints. Hints appear to allocate ByteBuffers indefinitely so
they can't provide backpressure based on available disk capacity/throughput
anymore. Hints are submitted directly from the contributing thread so there is
no such thing as in-flight hint anymore. I'll clean up the existing hint
overload protection so that it kicks in based on number of pending buffers and
see where that gets me.
I am guessing that in practice hint overload is going to be a lot harder to hit
now that it is based on flat files. Clearly sequential IO performance outstrips
the ability to process even large incoming mutations.
> Bound the number of in-flight requests at the coordinator
> ---------------------------------------------------------
>
> Key: CASSANDRA-9318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9318
> Project: Cassandra
> Issue Type: Improvement
> Components: Local Write-Read Paths, Streaming and Messaging
> Reporter: Ariel Weisberg
> Assignee: Ariel Weisberg
> Fix For: 2.1.x, 2.2.x
>
>
> It's possible to somewhat bound the amount of load accepted into the cluster
> by bounding the number of in-flight requests and request bytes.
> An implementation might do something like track the number of outstanding
> bytes and requests and if it reaches a high watermark disable read on client
> connections until it goes back below some low watermark.
> Need to make sure that disabling read on the client connection won't
> introduce other issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
