[jira] [Commented] (CASSANDRA-14281) Reduce contention on DecayingEstimatedHistogramReservoir
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391327#comment-16391327 ] Michael Burman commented on CASSANDRA-14281: Here's a branch for these updates: [https://github.com/burmanm/cassandra/tree/latency_metrics] Benchmark results (on my laptop, which doesn't have a lot of contention) trunk: [java] LatencyTrackingBench.benchInsertToDEHR thrpt 5 5509872.821 ± 523366.638 ops/s [java] LatencyTrackingBench.benchLatencyMetricsWrite thrpt 5 2148740.763 ± 839102.027 ops/s branch: [java] LatencyTrackingBench.benchInsertToDEHR thrpt 5 16387458.999 ± 9339490.676 ops/s [java] LatencyTrackingBench.benchLatencyMetricsWrite thrpt 5 6789997.113 ± 1778364.870 ops/s So around 3 times the performance. 13% of the time is only spent in DEHR in the benchmark, benchmark-related cleaning takes ~10% and GC around 22%. Around 42% in the Dropwizard inherited (Counter & Timer's other work) classes. Significant improvements to these would require removing Dropwizard classes from LatencyMetrics (and use of ThreadLocal etc to reduce volatile reads, CAS writes etc) > Reduce contention on DecayingEstimatedHistogramReservoir > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14281) Reduce contention on DecayingEstimatedHistogramReservoir
[
https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390824#comment-16390824
]
Michael Burman commented on CASSANDRA-14281:
I'm not sure how LongAdder would change the need for merging? The overhead of
updating multiple LatencyMetrics is not reduced (this is what the merging was
created for) that way (multiple LongAdder updates are not free either).
LongAdder does use a CAS method in updates though (thus it still prevents some
out-of-order behavior), since it does hash a thread-id to a slot (it isn't
ThreadLocal). There's also a small added concurrency issues when rescaling with
LongAdders, given the following code:
{code:java}
for(int i = 0; i < adders.length; i++) {
long storedValue = adders[i].sumThenReset();
storedValue = Math.round(storedValue / rescaleFactor);
adders[i].add(storedValue);
}
{code}
Since the reset() in LongAdder is not concurrency-safe. Although if acceptable
risk, I can do that too.
> Reduce contention on DecayingEstimatedHistogramReservoir
>
>
> Key: CASSANDRA-14281
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14281
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Burman
>Assignee: Michael Burman
>Priority: Major
>
> Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each
> update operation, which causes a contention if there are more than one thread
> updating the histogram. This impacts scalability when using larger machines.
> We should make it lock-free as much as possible and also avoid a single
> CAS-update from blocking all the concurrent threads from making an update.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14281) Reduce contention on DecayingEstimatedHistogramReservoir
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390310#comment-16390310 ] Chris Lohfink commented on CASSANDRA-14281: --- Is there a reason to not use an array of longadders which keeps the long for each bucket in a ThreadLocal than recreate the logic to do merging on histogram level? Would be a smaller change to just change AtomicLong[]s to LongAdder[] > Reduce contention on DecayingEstimatedHistogramReservoir > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14281) Reduce contention on DecayingEstimatedHistogramReservoir
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389579#comment-16389579 ] Michael Burman commented on CASSANDRA-14281: Updated the patch to include merging two reservoir snapshots (with certain preassumptions given the use-case inside Cassandra I was intending the feature to be used for) [https://github.com/burmanm/cassandra/commit/c1303f493510e40a4f3d05af74e26d0cdc1c4ea6] For contention reductions (I'm not going to reduce them more right now) aggregating would allow using ThreadLocal instances of LatencyMetrics, but at that point we should also get rid of the CAS operations. > Reduce contention on DecayingEstimatedHistogramReservoir > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14281) Reduce contention on DecayingEstimatedHistogramReservoir
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384622#comment-16384622 ] Michael Burman commented on CASSANDRA-14281: Well, there's certainly still too much contention. The performance does not really scale with 16 cores machine. > Reduce contention on DecayingEstimatedHistogramReservoir > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14281) Reduce contention on DecayingEstimatedHistogramReservoir
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384570#comment-16384570 ] Michael Burman commented on CASSANDRA-14281: I've changed the ticket description to match the discussion here. The reduced amount of histograms updates during writes should be separate work. > Reduce contention on DecayingEstimatedHistogramReservoir > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
