[jira] [Comment Edited] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451594#comment-16451594 ] Chris Lohfink edited comment on CASSANDRA-14281 at 4/25/18 3:43 AM: yeah i +1d the code on 18th, the benchmark was just followup me trying it out to see same improvement he did, which I didn't see on my laptop. Main feedback (was in IRC so not in comments, adding it now for posterity) that he fixed was children metrics tracking parent to prevent negative values after table drops was (Author: cnlwsu): yeah i +1d the code on 18th, the benchmark was just followup me trying it out to see same improvement he did, which I didn't see on my laptop. Main feedback (was in IRC so not in comments, adding it now for posterity) that he fixed was parent metrics tracking children to prevent negative values after table drops > Improve LatencyMetrics performance by reducing write path processing > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > Attachments: bench.png, bench2.png, benchmark.html, benchmark2.png > > > Currently for each write/read/rangequery/CAS touching the CFS we write a > latency metric which takes a lot of processing time (up to 66% of the total > processing time if the update was empty). > The way latencies are recorded is to use both a dropwizard "Timer" as well as > "Counter". Latter is used for totalLatency and the previous is decaying > metric for rates and certain percentile metrics. We then replicate all of > these CFS writes to the KeyspaceMetrics and globalWriteLatencies. > Instead of doing this on the write phase we should merge the metrics when > they're read. This is much less common occurrence and thus we save a lot of > CPU time in total. This also speeds up the write path. > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427938#comment-16427938 ] Michael Shuler edited comment on CASSANDRA-14281 at 4/6/18 4:17 AM: !bench2.png|width=700,height=400! Converted the TIFF bench.png to PNG was (Author: mshuler): !bench2.png! Converted the TIFF bench.png to PNG > Improve LatencyMetrics performance by reducing write path processing > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > Attachments: bench.png, bench2.png > > > Currently for each write/read/rangequery/CAS touching the CFS we write a > latency metric which takes a lot of processing time (up to 66% of the total > processing time if the update was empty). > The way latencies are recorded is to use both a dropwizard "Timer" as well as > "Counter". Latter is used for totalLatency and the previous is decaying > metric for rates and certain percentile metrics. We then replicate all of > these CFS writes to the KeyspaceMetrics and globalWriteLatencies. > Instead of doing this on the write phase we should merge the metrics when > they're read. This is much less common occurrence and thus we save a lot of > CPU time in total. This also speeds up the write path. > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427874#comment-16427874 ] Chris Lohfink edited comment on CASSANDRA-14281 at 4/6/18 3:15 AM: --- I dont see any issues with code and thats an improvement certainly. I tried cassandra-stress locally and !bench.png|width=799,height=417! didnt get an improvement but it maybe different with a faster host (tried 8-128 threads in powers of 2) was (Author: cnlwsu): I dont see any issues with code and thats an improvement certainly. I tried cassandra-stress locally and !image-2018-04-05-22-06-05-947.png|width=799,height=417! didnt get an improvement but it maybe different with a faster host (tried 8-128 threads in powers of 2) > Improve LatencyMetrics performance by reducing write path processing > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > Attachments: bench.png > > > Currently for each write/read/rangequery/CAS touching the CFS we write a > latency metric which takes a lot of processing time (up to 66% of the total > processing time if the update was empty). > The way latencies are recorded is to use both a dropwizard "Timer" as well as > "Counter". Latter is used for totalLatency and the previous is decaying > metric for rates and certain percentile metrics. We then replicate all of > these CFS writes to the KeyspaceMetrics and globalWriteLatencies. > Instead of doing this on the write phase we should merge the metrics when > they're read. This is much less common occurrence and thus we save a lot of > CPU time in total. This also speeds up the write path. > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427874#comment-16427874 ] Chris Lohfink edited comment on CASSANDRA-14281 at 4/6/18 3:12 AM: --- I dont see any issues with code and thats an improvement certainly. I tried cassandra-stress locally and !image-2018-04-05-22-06-05-947.png|width=799,height=417! didnt get an improvement but it maybe different with a faster host (tried 8-128 threads in powers of 2) was (Author: cnlwsu): I dont see any issues with code and thats an improvement certainly. I tried cassandra-stress locally and !image-2018-04-05-22-06-05-947.png|width=799,height=417! didnt get an improvement but it maybe different with a faster host (tried 8-128 threads in powers of 2) > Improve LatencyMetrics performance by reducing write path processing > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > Attachments: image-2018-04-05-22-06-05-947.png > > > Currently for each write/read/rangequery/CAS touching the CFS we write a > latency metric which takes a lot of processing time (up to 66% of the total > processing time if the update was empty). > The way latencies are recorded is to use both a dropwizard "Timer" as well as > "Counter". Latter is used for totalLatency and the previous is decaying > metric for rates and certain percentile metrics. We then replicate all of > these CFS writes to the KeyspaceMetrics and globalWriteLatencies. > Instead of doing this on the write phase we should merge the metrics when > they're read. This is much less common occurrence and thus we save a lot of > CPU time in total. This also speeds up the write path. > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
