Hi Pierre,

Is there anything stopping you from using the compaction_throughput <https://github.com/apache/cassandra/blob/f9e033f519c14596da4dc954875756a69aea4e78/conf/cassandra.yaml#L989> option in the cassandra.yaml file to manage the performance impact of compaction operations?

With thread priority, there's a failure scenario on busy nodes when the read operations uses too much CPU. If the compaction thread has lower priority, it does not get enough CPU time to run, and SSTable files will build up, causing read to become slower and more expensive, which in turn result in compaction gets even less CPU time. At the end, one of the following three will happen:

 * the node becomes too slow and most queries time out
 * the Java process crashes due to too many open files or OOM because
   JVM GC can't keep up
 * the filesystem run out of free space or inodes

However, I'm unsure whether the compaction thread priority was intentionally removed from 4.1.0. Someone familiar with this matter may be able to answer that.


On 22/02/2024 13:26, Pierre Fersing wrote:

Hello all,

I've recently upgraded to Cassandra 4.1 and see a change in compaction behavior that seems unwanted:

* With Cassandra 3.11 compaction was run by thread in low priority and thus using CPU nice (visible using top) (I believe Cassandra 4.0 also had this behavior)

* With Cassandra 4.1, compactions are no longer run as low priority thread (compaction now use "normal" CPU).

This means that when the server had limited CPU, Cassandra compaction now compete for the CPU with other process (probably including Cassandra itself) that need CPU. When it was using CPU nice, the compaction only competed for CPU with other lower priority process which was great as it leaves CPU available for processes that need to kept small response time (like an API used by human).

Is it wanted to lose this feature in Cassandra 4.1 or was it just a forget during re-write of compaction executor ? Should I open a bug to re-introduce this feature in Cassandra ?

I've done few searches, and:

* I believe compaction used CPU nice because the compactor executor was created with minimal priority: https://github.com/apache/cassandra/blob/cassandra-3.11.16/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1906 <https://github.com/apache/cassandra/blob/cassandra-3.11.16/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1906>

* I think it was dropped by commit https://github.com/apache/cassandra/commit/be1f050bc8c0cd695a42952e3fc84625ad48d83a <https://github.com/apache/cassandra/commit/be1f050bc8c0cd695a42952e3fc84625ad48d83a>

* It looks doable to set the thread priority with new executor, I think adding ".withThreadPriority(Thread.MIN_PRIORITY)" when using executorFactory in https://github.com/apache/cassandra/blob/77a3e0e818df3cce45a974ecc977ad61bdcace47/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L2028 <https://github.com/apache/cassandra/blob/77a3e0e818df3cce45a974ecc977ad61bdcace47/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L2028>should do it.

Did I miss a reason to no longer use low priority threads for compaction ? Should I open a bug for re-adding this feature / submit a PR ?


Pierre Fersing

Reply via email to