[
https://issues.apache.org/jira/browse/CASSANDRA-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104145#comment-18104145
]
Dmitry Konstantinov edited comment on CASSANDRA-21569 at 8/12/26 4:43 PM:
--------------------------------------------------------------------------
We also have an option to change a bit a semantic of
org.apache.cassandra.transport.ServerConnection#requests metric: now it is
about number of requests received from a client and sent to process (so we do
not count cases when we reject the request before a particular step due to an
error, backpressure, etc). If we can count all the cases (what is probably even
better from observability point of view...) and increment the counter earlier
then we can move it to a moment before switching to Dispatcher#requestExecutor
pool and do it in Netty thread. In such case we can use the fact you mentioned
about Netty thread ownership for a connection, which means what only single
Netty thread can update the metric and we can just use AtomicLong/volatile long
or setRelease/getAcquire
This way is not only remove the write issue and makes reads very cheap but it
also would reduce memory usage if we consider the cases with many connections.
So far, it looks like the best option for me..
was (Author: dnk):
We also have an option to change a bit a semantic of
org.apache.cassandra.transport.ServerConnection#requests metric: now it is
about number of requests received from a client and sent to process (so we do
not count cases when we reject the request before a particular step due to an
error, backpressure, etc). If we can count all the cases (what is probably even
better from observability point of view...) and increment the counter earlier
then we can move it to a moment before switching to Dispatcher#requestExecutor
pool and do it in Netty thread. In such case we can use the fact you mentioned
about Netty thread ownership for a connection, which means what only single
Netty thread can update the metric and we can just use AtomicLong/volatile long
or setRelease/getAcquire
> Revert ServerConnection.requests from ThreadLocalCounter to Dropwizard Counter
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-21569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21569
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
> Reporter: Francisco Guerrero
> Assignee: Francisco Guerrero
> Priority: Normal
> Attachments: ThreadLocalMetricsChurnBench.java,
> ThreadLocalMetricsReadScanBench.java, may17_allread_cpu.html
>
>
> h2. Summary
> {{ServerConnection.requests}} was changed from a Dropwizard {{Counter}}
> ({{LongAdder}}) to {{ThreadLocalCounter}} in CASSANDRA-21400. I would like to
> reconsider this change: it gains a negligible write speedup while making
> {{system_views.clients}} scans and metric reads under connection churn
> dramatically more expensive. I propose we revert *only this one site*.
> h2. The trade-off to consider
> {{ThreadLocalCounter}}'s only advantage is a cheaper {{inc()}} (a
> thread-local array bump vs a {{LongAdder}} cell). Its cost is that
> {{getCount()}} takes a process-wide shared read lock and sums the counter's
> slot across *every live thread*, and each counter close
> ({{recycleMetricId()}}) takes that lock's *write* side and scans every live
> thread. That trade only pays off for a small, fixed set of write-hot
> counters. {{ServerConnection.requests}} is the opposite:
> * *Benefit is negligible here.* It is incremented once per request by the
> single Netty thread that owns the connection (no contention), so
> {{LongAdder}} stays on its base cell. Measured delta is ~1-2 ns per increment
> (well under 0.1% of CQL request processing).
> * *Cost is large here.* It is per-connection (high cardinality, churny) and
> is read once-per-row by {{ClientsTable}} ({{system_views.clients}} /
> {{nodetool clientstats}}), so a single query becomes an O(connections x
> live-threads) scan under the shared lock, and connection open/close churn
> generates write-lock traffic that stalls reads of unrelated counters.
> h2. Evidence (JMH; single fork, JDK 11 -- read the ratios, not absolute
> numbers)
> *Write* (the benefit) -- per-increment cost is ~1-2 ns for a single
> low-contention counter; even the worst measured case (cache-polluted, 100
> counters) is ~35 ns/inc.
> *Read* (the cost) -- one {{system_views.clients}} scan, us/op:
> || Connections || Live threads || LongAdder || ThreadLocalCounter ||
> | 10,000 | 64 | 9.2 | 455 |
> | 10,000 | 256 | 9.6 | 6,786 |
> | 50,000 | 256 | 57.3 | 120,757 |
> {{LongAdder}} is flat in thread count; {{ThreadLocalCounter}} scales with
> connections x threads (~121 ms per scan at 50k connections / 256 threads).
> *Churn* (the cost) -- read throughput of a stable counter while N threads
> close counters, at 256 live threads: {{0 -> 10,667 ops/ms}}, {{1 -> 1,384
> (-7.7x)}}, {{4 -> 509 (-21x)}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
