Francisco Guerrero created CASSANDRA-21569:
----------------------------------------------
Summary: Revert ServerConnection.requests from ThreadLocalCounter
to Dropwizard Counter
Key: CASSANDRA-21569
URL: https://issues.apache.org/jira/browse/CASSANDRA-21569
Project: Apache Cassandra
Issue Type: Bug
Reporter: Francisco Guerrero
Assignee: Francisco Guerrero
h2. Summary
{{ServerConnection.requests}} was changed from a Dropwizard {{Counter}}
({{LongAdder}}) to {{ThreadLocalCounter}} in CASSANDRA-21400. I would like to
reconsider this change: it gains a negligible write speedup while making
{{system_views.clients}} scans and metric reads under connection churn
dramatically more expensive. I propose we revert *only this one site*.
h2. The trade-off to consider
{{ThreadLocalCounter}}'s only advantage is a cheaper {{inc()}} (a thread-local
array bump vs a {{LongAdder}} cell). Its cost is that {{getCount()}} takes a
process-wide shared read lock and sums the counter's slot across *every live
thread*, and each counter close ({{recycleMetricId()}}) takes that lock's
*write* side and scans every live thread. That trade only pays off for a small,
fixed set of write-hot counters. {{ServerConnection.requests}} is the opposite:
* *Benefit is negligible here.* It is incremented once per request by the
single Netty thread that owns the connection (no contention), so {{LongAdder}}
stays on its base cell. Measured delta is ~1-2 ns per increment (well under
0.1% of CQL request processing).
* *Cost is large here.* It is per-connection (high cardinality, churny) and is
read once-per-row by {{ClientsTable}} ({{system_views.clients}} / {{nodetool
clientstats}}), so a single query becomes an O(connections x live-threads) scan
under the shared lock, and connection open/close churn generates write-lock
traffic that stalls reads of unrelated counters.
h2. Evidence (JMH; single fork, JDK 11 -- read the ratios, not absolute numbers)
*Write* (the benefit) -- per-increment cost is ~1-2 ns for a single
low-contention counter; even the worst measured case (cache-polluted, 100
counters) is ~35 ns/inc.
*Read* (the cost) -- one {{system_views.clients}} scan, us/op:
|| Connections || Live threads || LongAdder || ThreadLocalCounter ||
| 10,000 | 64 | 9.2 | 455 |
| 10,000 | 256 | 9.6 | 6,786 |
| 50,000 | 256 | 57.3 | 120,757 |
{{LongAdder}} is flat in thread count; {{ThreadLocalCounter}} scales with
connections x threads (~121 ms per scan at 50k connections / 256 threads).
*Churn* (the cost) -- read throughput of a stable counter while N threads close
counters, at 256 live threads: {{0 -> 10,667 ops/ms}}, {{1 -> 1,384 (-7.7x)}},
{{4 -> 509 (-21x)}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
