[jira] [Commented] (CASSANDRA-16161) Validation Compactions causing Java GC pressure
[ https://issues.apache.org/jira/browse/CASSANDRA-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234978#comment-17234978 ] Michael Semb Wever commented on CASSANDRA-16161: Thanks for the (long) unit tests in the patch [~stefan.miklosovic]! CI run is [here|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/219/pipeline]. > Validation Compactions causing Java GC pressure > --- > > Key: CASSANDRA-16161 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16161 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Local/Config, Tool/nodetool >Reporter: Cameron Zemek >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 3.11.x > > Attachments: 16161.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Validation Compactions are not rate limited which can cause Java GC pressure > and result in spikes in latency. > PR https://github.com/apache/cassandra/pull/814 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16161) Validation Compactions causing Java GC pressure
[ https://issues.apache.org/jira/browse/CASSANDRA-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226704#comment-17226704 ] Stefan Miklosovic commented on CASSANDRA-16161: --- https://github.com/apache/cassandra/pull/809 > Validation Compactions causing Java GC pressure > --- > > Key: CASSANDRA-16161 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16161 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Local/Config, Tool/nodetool >Reporter: Cameron Zemek >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 3.11.x > > Attachments: 16161.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Validation Compactions are not rate limited which can cause Java GC pressure > and result in spikes in latency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16161) Validation Compactions causing Java GC pressure
[ https://issues.apache.org/jira/browse/CASSANDRA-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226450#comment-17226450 ] Stefan Miklosovic commented on CASSANDRA-16161: --- Jus to make it clear, we are fixing and focusin on 3.11.x only, right? We are fine with this, our primary concern is to make it happen in 3.11.x, but anyway, we are patching it on our side regardless so no rush. I ve briefly looked and validation compaction logic is rewritten a bit in 4.0 and this stuff does not make sense there anymore ... > Validation Compactions causing Java GC pressure > --- > > Key: CASSANDRA-16161 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16161 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Local/Config, Tool/nodetool >Reporter: Cameron Zemek >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 3.11.x, 3.11.8 > > Attachments: 16161.patch > > > Validation Compactions are not rate limited which can cause Java GC pressure > and result in spikes in latency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16161) Validation Compactions causing Java GC pressure
[
https://issues.apache.org/jira/browse/CASSANDRA-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224671#comment-17224671
]
Alexander Dejanovski commented on CASSANDRA-16161:
--
My initial preference while reading this ticket was to use different throttles
as well. As mentioned by [~mck], validation compactions put a slightly
different type of pressure on nodes and folks might want to unthrottle
validation compactions (current behavior) and keep a throttle on compactions.
But on the other hand, disks are now way faster than they used to be when
compaction throttle was introduced and heap pressure is mostly what we're
protecting the clusters from I guess. We used to consider at TLP that over
45MB/s Cassandra couldn't keep up anyway in compaction sustained throughput
because of heap pressure (it would only allow bursts). This will indeed change
in 4.0 with all the improvements made to lower compaction heap pressure.
I also think that repair should be throttled in order to lighten its impact on
clusters and force folks to investigate if it's not going fast enough, rather
than harm clusters when using defaults to make it go fast.
Also, adding a new hidden configuration setting just for 3.0/3.x this close to
4.0 going GA doesn't seem like the best thing to do.
TL;DR: +1 on using {{compaction_throughput_mb_per_sec}} to throttle validation
compactions as well as standard compactions.
> Validation Compactions causing Java GC pressure
> ---
>
> Key: CASSANDRA-16161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16161
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Compaction, Local/Config, Tool/nodetool
>Reporter: Cameron Zemek
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 3.11.x, 3.11.8
>
> Attachments: 16161.patch
>
>
> Validation Compactions are not rate limited which can cause Java GC pressure
> and result in spikes in latency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16161) Validation Compactions causing Java GC pressure
[
https://issues.apache.org/jira/browse/CASSANDRA-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224483#comment-17224483
]
Michael Semb Wever commented on CASSANDRA-16161:
bq. This could almost be classified as a bug. In conf/cassandra.yaml it states
for compaction_throughput_mb_per_sec: "Note that this account for all types of
compaction, including validation compaction"
I am keen on making this a bug and throttling validation compactions under
{{compaction_throughput_mb_per_sec}}, as the existing documentation says it
already does. But have some questions, like some of the past discussion in
CASSANDRA-3635. There are differences between clusters on how validation and
normal compactions interact, with each using resources differently: validations
use more cpu and only read IO compared to normal compactions.
If validation compactions are made to honour
{{compaction_throughput_mb_per_sec}}, what impact might that have on existing
clusters? (particularly those unsuspecting to such a change in a patch
upgrade). If there are users that are dependent on validation compactions being
unthrottled and completing faster than other compactions, how would they
achieve that after this change?
> Validation Compactions causing Java GC pressure
> ---
>
> Key: CASSANDRA-16161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16161
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Compaction, Local/Config, Tool/nodetool
>Reporter: Cameron Zemek
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 3.11.x, 3.11.8
>
> Attachments: 16161.patch
>
>
> Validation Compactions are not rate limited which can cause Java GC pressure
> and result in spikes in latency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16161) Validation Compactions causing Java GC pressure
[ https://issues.apache.org/jira/browse/CASSANDRA-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213512#comment-17213512 ] Cameron Zemek commented on CASSANDRA-16161: --- This could almost be classified as a bug. In conf/cassandra.yaml is states for compaction_throughput_mb_per_sec: "Note that this account for all types of compaction, including validation compaction" Which makes me lean to having a shared throttle. > Validation Compactions causing Java GC pressure > --- > > Key: CASSANDRA-16161 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16161 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction, Local/Config, Tool/nodetool >Reporter: Cameron Zemek >Assignee: Cameron Zemek >Priority: Normal > Fix For: 3.11.x, 3.11.8 > > Attachments: 16161.patch > > > Validation Compactions are not rate limited which can cause Java GC pressure > and result in spikes in latency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16161) Validation Compactions causing Java GC pressure
[ https://issues.apache.org/jira/browse/CASSANDRA-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17206225#comment-17206225 ] Chris Lohfink commented on CASSANDRA-16161: --- While tangental to adding a throttle, offheap merkle trees and some of the perf improvements (ie xor impl and more aggressive dereferencing) has helped a lot as well (CASSANDRA-15202). Its a lot better in 4.0 from my experiences. For 3.x though I agree that validation compaction is brutal and this could provide a workaround to keep functioning. Only question I think is if the compaction throughput throttle should be shared vs creating a new one but I am happy with this approach. The code looks straight forward, +1 > Validation Compactions causing Java GC pressure > --- > > Key: CASSANDRA-16161 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16161 > Project: Cassandra > Issue Type: Improvement >Reporter: Cameron Zemek >Priority: Normal > Fix For: 3.11.x, 3.11.8 > > Attachments: 16161.patch > > > Validation Compactions are not rate limited which can cause Java GC pressure > and result in spikes in latency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16161) Validation Compactions causing Java GC pressure
[ https://issues.apache.org/jira/browse/CASSANDRA-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17205936#comment-17205936 ] Cameron Zemek commented on CASSANDRA-16161: --- In order to not break existing behavior I have made validation compactions have their own separate throttle in this patch. However for 4.x I think it make more sense to reuse the some rate limiter that is on normal compactions. > Validation Compactions causing Java GC pressure > --- > > Key: CASSANDRA-16161 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16161 > Project: Cassandra > Issue Type: Improvement >Reporter: Cameron Zemek >Priority: Normal > Fix For: 3.11.x > > Attachments: 16161.patch > > > Validation Compactions are not rate limited which can cause Java GC pressure > and result in spikes in latency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
