[
https://issues.apache.org/jira/browse/CASSANDRA-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342344#comment-17342344
]
Benedict Elliott Smith commented on CASSANDRA-16663:
----------------------------------------------------
We should also look at propagating internode back-pressure to clients in 4.0.
I wonder if that, combined with a client rate-limit, would be enough to
deprecate the existing bytes-in-flight per client. Or at least we should look
to see if we can rationalise them into a coherent system, with a unified API
for configuration. It's already a lot of different limits.
> Request-Based Native Transport Rate-Limiting
> --------------------------------------------
>
> Key: CASSANDRA-16663
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16663
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
> Reporter: Caleb Rackliffe
> Assignee: Caleb Rackliffe
> Priority: Normal
> Fix For: 4.0.x, 4.x
>
>
> Together, CASSANDRA-14855, CASSANDRA-15013, and CASSANDRA-15519 added support
> for a runtime-configurable, per-coordinator limit on the number of bytes
> allocated for concurrent requests over the native protocol. It supports
> channel back-pressure by default, and optionally supports throwing
> OverloadedException if that is requested in the relevant connection’s STARTUP
> message.
> This can be an effective tool to prevent the coordinator from running out of
> memory, but it may not correspond to how expensive a queries are or provide a
> direct conceptual mapping to how users think about request capacity. I
> propose adding the option of request-based (or perhaps more correctly
> message-based) back-pressure, coexisting with (and reusing the logic that
> supports) the current bytes-based back-pressure.
> _We can roll this forward in phases_, where the server’s cost accounting
> becomes more accurate, we segment limits by operation type/keyspace/etc., and
> the client/driver reacts more intelligently to (especially non-back-pressure)
> overload, _but something minimally viable could look like this_:
> 1.) Reuse most of the existing logic in Limits, et al. to support a simple
> per-coordinator limit only on native transport requests in flight (i.e. where
> requests have a fixed cost of 1). Under this limit will be CQL reads and
> writes, but also auth requests, prepare requests, and batches. This is
> obviously simplistic, and it does not account for the variation in cost
> between individual queries, but even a fixed cost model should be useful in
> aggregate.
> * If the client specifies THROW_ON_OVERLOAD in its STARTUP message at
> connection time, a breach of the per-node limit will result in an
> OverloadedException being propagated to the client, and the server will
> discard the request.
> * If THROW_ON_OVERLOAD is not specified, the server will stop consuming
> messages from the channel/socket, which should back-pressure the client,
> while the message continues to be processed.
> 2.) This limit is infinite by default, and can be enabled via the YAML config
> or JMX at runtime. (The literal default of -1 corresponding to “no limit” can
> be used here.)
> 3.) The current value of the limit is available via JMX, and metrics around
> coordinator operations/second are already available to compare against it.
> 4.) Any interaction with existing byte-based limits will intersect. (i.e. A
> breach of any limit, bytes or request-based, will actuate back-pressure or
> OverloadedExceptions.)
> In this first pass, explicitly out of scope would be any work on the
> client/driver side.
> In terms of validation/testing, our biggest concern with anything that adds
> overhead on a very hot path is performance. In particular, we want to fully
> understand how the client and server perform along two axes constituting 4
> scenarios. Those are a.) whether or not we are breaching the request limit
> and b.) whether the server is throwing on overload at the behest of the
> client. Having said that, query execution should dwarf the cost of limit
> accounting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
