Re: Help determining pending compactions

[Richard Hesse]

Thanks for the tip Eric. We're actually on 3.2 and the issue isn't with the
Reaper. The issue is with Cassandra. It will report that a table has
pending compactions, but it will never actually start compacting. The
pending number stays at that level until we run a manual compaction.

-richard


On Mon, Nov 7, 2022 at 8:53 AM Eric Ferrenbach <
eric.ferrenb...@milliporesigma.com> wrote:

> We had issues where Reaper would never actually start some repairs.  The
> GUI would say RUNNING but the progress would be 0/.
>
> Datastax support said there is a bug and recommended upgrading to 3.2.
>
> Upgrading Reaper to 3.2 resolved our issue.
>
>
>
> Hope this helps.
>
> Eric
>
>
>
>
>
>
>
> *From:* Richard Hesse 
> *Sent:* Sunday, October 30, 2022 12:07 PM
> *To:* user@cassandra.apache.org
> *Subject:* Help determining pending compactions
>
>
>
> [WARNING - EXTERNAL EMAIL] Do not open links or attachments unless you
> recognize the sender of this email. If you are unsure please click the
> button "Report suspicious email"
>
>
>
> Hi, I'm hoping to get some help with a vexing issue with one of our
> keyspaces. During Reaper repair sessions, one keyspace will end up with
> hanging, non-started compactions. That is, the number of compactions as
> reported by nodetool compactionstats stays flat and there are no running
> compactions. Is there a way to determine which tables Cassandra is stuck on
> here?
>
>
>
> Looking at graphs of pending compactions during Reaper sessions, the
> number of compactions will shoot up (as expected). The number will work its
> way down, and sometimes it will stop and plateau at a fixed level. A full
> compaction will get things going again, but we prefer to avoid those (even
> with the -s option).
>
>
>
> I realize there are various compaction tuning parameters regarding minimum
> age, tombstone percentage, etc but I need to know which sstables to look at
> first before blindly changing values. These are leveled compaction strategy
> tables FWIW.
>
>
>
> TIA!
>
>
>
> -richard
>
>
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient,
> you must not copy this message or attachment or disclose the contents to
> any other person. If you have received this transmission in error, please
> notify the sender immediately and delete the message and any attachment
> from your system. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not accept liability for any omissions or errors in this
> message which may arise as a result of E-Mail-transmission or for damages
> resulting from any unauthorized changes of the content of this message and
> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not guarantee that this message is free of viruses and does
> not accept liability for any damages caused by any virus transmitted
> therewith.
>
>
>
> Click emdgroup.com/disclaimer
>  to
> access the German, French, Spanish, Portuguese, Turkish, Polish and Slovak
> versions of this disclaimer.
>
>
>
> Please find our Privacy Statement information by clicking here: 
> emdgroup.com/privacy-statement
> (U.S.)  or 
> emdserono.com/privacy-statement
> (Canada) 
>

RE: Help determining pending compactions

[Eric Ferrenbach]

We had issues where Reaper would never actually start some repairs.  The GUI 
would say RUNNING but the progress would be 0/.
Datastax support said there is a bug and recommended upgrading to 3.2.
Upgrading Reaper to 3.2 resolved our issue.

Hope this helps.
Eric



From: Richard Hesse 
Sent: Sunday, October 30, 2022 12:07 PM
To: user@cassandra.apache.org
Subject: Help determining pending compactions


[WARNING - EXTERNAL EMAIL] Do not open links or attachments unless you 
recognize the sender of this email. If you are unsure please click the button 
"Report suspicious email"

Hi, I'm hoping to get some help with a vexing issue with one of our keyspaces. 
During Reaper repair sessions, one keyspace will end up with hanging, 
non-started compactions. That is, the number of compactions as reported by 
nodetool compactionstats stays flat and there are no running compactions. Is 
there a way to determine which tables Cassandra is stuck on here?

Looking at graphs of pending compactions during Reaper sessions, the number of 
compactions will shoot up (as expected). The number will work its way down, and 
sometimes it will stop and plateau at a fixed level. A full compaction will get 
things going again, but we prefer to avoid those (even with the -s option).

I realize there are various compaction tuning parameters regarding minimum age, 
tombstone percentage, etc but I need to know which sstables to look at first 
before blindly changing values. These are leveled compaction strategy tables 
FWIW.

TIA!

-richard



This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, you 
must not copy this message or attachment or disclose the contents to any other 
person. If you have received this transmission in error, please notify the 
sender immediately and delete the message and any attachment from your system. 
Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept 
liability for any omissions or errors in this message which may arise as a 
result of E-Mail-transmission or for damages resulting from any unauthorized 
changes of the content of this message and any attachment thereto. Merck KGaA, 
Darmstadt, Germany and any of its subsidiaries do not guarantee that this 
message is free of viruses and does not accept liability for any damages caused 
by any virus transmitted therewith.



Click 
emdgroup.com/disclaimer
 to access the German, French, Spanish, Portuguese, Turkish, Polish and Slovak 
versions of this disclaimer.



Please find our Privacy Statement information by clicking here: 
emdgroup.com/privacy-statement 
(U.S.) or 
emdserono.com/privacy-statement 
(Canada)

Re: Help determining pending compactions

[Richard Hesse]

Sorry about that. 4.0.6

On Sun, Oct 30, 2022, 11:19 AM Dinesh Joshi  wrote:

> It would be helpful if you could tell us what version of Cassandra you’re
> using?
>
> Dinesh
>
> > On Oct 30, 2022, at 10:07 AM, Richard Hesse  wrote:
> >
> > 
> > Hi, I'm hoping to get some help with a vexing issue with one of our
> keyspaces. During Reaper repair sessions, one keyspace will end up with
> hanging, non-started compactions. That is, the number of compactions as
> reported by nodetool compactionstats stays flat and there are no running
> compactions. Is there a way to determine which tables Cassandra is stuck on
> here?
> >
> > Looking at graphs of pending compactions during Reaper sessions, the
> number of compactions will shoot up (as expected). The number will work its
> way down, and sometimes it will stop and plateau at a fixed level. A full
> compaction will get things going again, but we prefer to avoid those (even
> with the -s option).
> >
> > I realize there are various compaction tuning parameters regarding
> minimum age, tombstone percentage, etc but I need to know which sstables to
> look at first before blindly changing values. These are leveled compaction
> strategy tables FWIW.
> >
> > TIA!
> >
> > -richard
>

Re: Help determining pending compactions

[Dinesh Joshi]

It would be helpful if you could tell us what version of Cassandra you’re using?

Dinesh

> On Oct 30, 2022, at 10:07 AM, Richard Hesse  wrote:
> 
> 
> Hi, I'm hoping to get some help with a vexing issue with one of our 
> keyspaces. During Reaper repair sessions, one keyspace will end up with 
> hanging, non-started compactions. That is, the number of compactions as 
> reported by nodetool compactionstats stays flat and there are no running 
> compactions. Is there a way to determine which tables Cassandra is stuck on 
> here?
> 
> Looking at graphs of pending compactions during Reaper sessions, the number 
> of compactions will shoot up (as expected). The number will work its way 
> down, and sometimes it will stop and plateau at a fixed level. A full 
> compaction will get things going again, but we prefer to avoid those (even 
> with the -s option).
> 
> I realize there are various compaction tuning parameters regarding minimum 
> age, tombstone percentage, etc but I need to know which sstables to look at 
> first before blindly changing values. These are leveled compaction strategy 
> tables FWIW.
> 
> TIA!
> 
> -richard