[
https://issues.apache.org/jira/browse/CASSANDRA-15013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806963#comment-16806963
]
Benedict commented on CASSANDRA-15013:
--------------------------------------
Thanks [~sumanth.pasupuleti] for the patch. I've only taken a quick glance at
it so far, but it looks pretty good, and I'm looking forward to integrating it.
There's only one substantive comment I have for the moment, which is that I
think when we {{setAutoread(false)}}, we need to not discard the message. We
should perform only one of the two options; if we ever discard, I think we
should always let the user know by throwing {{OverloadedException}}.
If we want to use back pressure without throwing away any messages that are
over our limit, I can think of two fairly straight forward mechanisms, with the
best being unfortunately quite difficult given our current Netty pipeline.
# Like we have done for CASSANDRA-15066, we would ideally leave the message
unparsed in the buffer until we have capacity to process it.
# Alternatively, as an easier solution, we can unconditionally enqueue the
message to the executor, assuming that we must have a fairly limited quantity
of bytes we can overshoot by
Also, as a point of consideration only, we _might_ also want to limit the
number of bytes we have in-flight per-endpoint, rather than per-channel, to
avoid a given host spamming the database with many connections. It's perhaps
not a very good unit to limit by, either, since an IP address may host many
applications, but an application can also open an arbitrary number of
connections, and crowd out all other hosts...
Just something to consider, I'm not sure what the best constraints are here.
Perhaps, similar to CASSANDRA-15066, a very small per-connection limit that can
always be consumed, to ensure progress on any given channel, with per-endpoint
and global limits for when these are exceeded (though this is less obvious than
for internode, as there could be many more clients connected, so it would be
plausible for these low limits to consume a great deal in combination).
It might be that we want to introduce the concept of an application identifier,
so that users who want to run multiple applications per host can do so, while
still ensuring QoS to other applications if one goes awry.
> Message Flusher queue can grow unbounded, potentially running JVM out of
> memory
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-15013
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15013
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client
> Reporter: Sumanth Pasupuleti
> Assignee: Sumanth Pasupuleti
> Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0, 3.0.x, 3.11.x
>
> Attachments: BlockedEpollEventLoopFromHeapDump.png,
> BlockedEpollEventLoopFromThreadDump.png, RequestExecutorQueueFull.png, heap
> dump showing each ImmediateFlusher taking upto 600MB.png
>
>
> This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue
> bounded, since, in the current state, items get added to the queue without
> any checks on queue size, nor with any checks on netty outbound buffer to
> check the isWritable state.
> We are seeing this issue hit our production 3.0 clusters quite often.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
