[
https://issues.apache.org/jira/browse/CASSANDRA-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103863#comment-18103863
]
Francisco Guerrero commented on CASSANDRA-21569:
------------------------------------------------
Similar to {{org.apache.cassandra.test.microbench.ThreadLocalMetricsBench}} I
created
{{org.apache.cassandra.test.microbench.ThreadLocalMetricsReadScanBench}} to
simulate a scan on system_views clients table. Attaching the bench files used
for getting these results.
> Revert ServerConnection.requests from ThreadLocalCounter to Dropwizard Counter
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-21569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21569
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
> Reporter: Francisco Guerrero
> Assignee: Francisco Guerrero
> Priority: Normal
> Attachments: ThreadLocalMetricsChurnBench.java,
> ThreadLocalMetricsReadScanBench.java
>
>
> h2. Summary
> {{ServerConnection.requests}} was changed from a Dropwizard {{Counter}}
> ({{LongAdder}}) to {{ThreadLocalCounter}} in CASSANDRA-21400. I would like to
> reconsider this change: it gains a negligible write speedup while making
> {{system_views.clients}} scans and metric reads under connection churn
> dramatically more expensive. I propose we revert *only this one site*.
> h2. The trade-off to consider
> {{ThreadLocalCounter}}'s only advantage is a cheaper {{inc()}} (a
> thread-local array bump vs a {{LongAdder}} cell). Its cost is that
> {{getCount()}} takes a process-wide shared read lock and sums the counter's
> slot across *every live thread*, and each counter close
> ({{recycleMetricId()}}) takes that lock's *write* side and scans every live
> thread. That trade only pays off for a small, fixed set of write-hot
> counters. {{ServerConnection.requests}} is the opposite:
> * *Benefit is negligible here.* It is incremented once per request by the
> single Netty thread that owns the connection (no contention), so
> {{LongAdder}} stays on its base cell. Measured delta is ~1-2 ns per increment
> (well under 0.1% of CQL request processing).
> * *Cost is large here.* It is per-connection (high cardinality, churny) and
> is read once-per-row by {{ClientsTable}} ({{system_views.clients}} /
> {{nodetool clientstats}}), so a single query becomes an O(connections x
> live-threads) scan under the shared lock, and connection open/close churn
> generates write-lock traffic that stalls reads of unrelated counters.
> h2. Evidence (JMH; single fork, JDK 11 -- read the ratios, not absolute
> numbers)
> *Write* (the benefit) -- per-increment cost is ~1-2 ns for a single
> low-contention counter; even the worst measured case (cache-polluted, 100
> counters) is ~35 ns/inc.
> *Read* (the cost) -- one {{system_views.clients}} scan, us/op:
> || Connections || Live threads || LongAdder || ThreadLocalCounter ||
> | 10,000 | 64 | 9.2 | 455 |
> | 10,000 | 256 | 9.6 | 6,786 |
> | 50,000 | 256 | 57.3 | 120,757 |
> {{LongAdder}} is flat in thread count; {{ThreadLocalCounter}} scales with
> connections x threads (~121 ms per scan at 50k connections / 256 threads).
> *Churn* (the cost) -- read throughput of a stable counter while N threads
> close counters, at 256 live threads: {{0 -> 10,667 ops/ms}}, {{1 -> 1,384
> (-7.7x)}}, {{4 -> 509 (-21x)}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
