Thanks for your detailed answers. I understand the reason why using low 
priority compaction may not be a great idea in the general case (the example 
with too high CPU for reading).

I’ll give a try with the compaction throughput which I total forgot that this 
option exists. It may fix the issue I see after my upgrade. I’ll see.
If it’s not fixing my trouble, I’ll try to confirm that my issue is indeed due 
to compaction putting too much pressure on CPU because going further as it will 
probably require more works that initial though and likely being a configurable 

Pierre Fersing

De : Dmitry Konstantinov <netud...@gmail.com>
Date : jeudi, 22 février 2024 à 20:39
À : user@cassandra.apache.org <user@cassandra.apache.org>
Objet : Re: Cassandra 4.1 compaction thread no longer low priority (cpu nice)
Hi all,

I was not participating in the changes but I analyzed the question some time 
ago from another side.
There were also changes related to -XX:ThreadPriorityPolicy JVM option. When 
you set a thread priority for a Java thread it does not mean it is always 
propagated as a native OS thread priority. To propagate the priority you should 
use  -XX:ThreadPriorityPolicy and -XX:JavaPriorityN_To_OSPriority JVM options, 
but there is an issue with them because JVM wants to be executed under root to 
set -XX:ThreadPriorityPolicy=1 which enables the priorities usage. A hack was 
invented long time ago to workaround it by setting -XX:ThreadPriorityPolicy=42 
value (any value not equal to 0 or 1) and bypass the not so needed and annoying 
grants validation logic (see 
for more details about).
It worked for Java 8 but then there was a change in Java 9 about adding extra 
validation for JVM option values (JEP 245: Validate JVM Command-Line Flag 
Arguments  - https://bugs.openjdk.org/browse/JDK-8059557) and the hack stopped 
to work and started to cause JVM failure with a validation error. As a reaction 
to it - the flag was removed from Cassandra JVM configuration files in 
https://issues.apache.org/jira/browse/CASSANDRA-13107. After it the lower 
priority value for compaction threads have not had any actual effect.
The interesting story is that the JVM logic has been changed to support the 
ability to set -XX:ThreadPriorityPolicy=1 for non-root users in Java 13 
(https://bugs.openjdk.org/browse/JDK-8215962) and the change was backported to 
Java 11 as well (https://bugs.openjdk.org/browse/JDK-8217494).
So, from this point of view I think it would be nice to return back the ability 
to set thread priority for compaction threads. At the same time I would not 
expect too much improvement by enabling it.

P.S. There was also an idea about using ionice 
(https://issues.apache.org/jira/browse/CASSANDRA-9946) but the current Linux IO 
schedulers do not take that into account anymore. It looks like the only 
scheduler that supported ionice was CFQ 
 and it was deprecated and removed since Linux kernel 5.x 


On Thu, 22 Feb 2024 at 15:30, Bowen Song via user 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>> wrote:

Hi Pierre,

Is there anything stopping you from using the 
 option in the cassandra.yaml file to manage the performance impact of 
compaction operations?

With thread priority, there's a failure scenario on busy nodes when the read 
operations uses too much CPU. If the compaction thread has lower priority, it 
does not get enough CPU time to run, and SSTable files will build up, causing 
read to become slower and more expensive, which in turn result in compaction 
gets even less CPU time. At the end, one of the following three will happen:

  *   the node becomes too slow and most queries time out
  *   the Java process crashes due to too many open files or OOM because JVM GC 
can't keep up
  *   the filesystem run out of free space or inodes

However, I'm unsure whether the compaction thread priority was intentionally 
removed from 4.1.0. Someone familiar with this matter may be able to answer 


On 22/02/2024 13:26, Pierre Fersing wrote:
Hello all,

I've recently upgraded to Cassandra 4.1 and see a change in compaction behavior 
that seems unwanted:

* With Cassandra 3.11 compaction was run by thread in low priority and thus 
using CPU nice (visible using top) (I believe Cassandra 4.0 also had this 

* With Cassandra 4.1, compactions are no longer run as low priority thread 
(compaction now use "normal" CPU).

This means that when the server had limited CPU, Cassandra compaction now 
compete for the CPU with other process (probably including Cassandra itself) 
that need CPU. When it was using CPU nice, the compaction only competed for CPU 
with other lower priority process which was great as it leaves CPU available 
for processes that need to kept small response time (like an API used by human).

Is it wanted to lose this feature in Cassandra 4.1 or was it just a forget 
during re-write of compaction executor ? Should I open a bug to re-introduce 
this feature in Cassandra ?

I've done few searches, and:

* I believe compaction used CPU nice because the compactor executor was created 
with minimal priority: 

* I think it was dropped by commit 

* It looks doable to set the thread priority with new executor, I think adding 
".withThreadPriority(Thread.MIN_PRIORITY)" when using executorFactory in 
 should do it.

Did I miss a reason to no longer use low priority threads for compaction ? 
Should I open a bug for re-adding this feature / submit a PR ?


Pierre Fersing

Dmitry Konstantinov

Reply via email to