Hi,
I've a production Cassandra cluster from 3.0.14 branch. Each node consists
of roughly 1.5TB data with a ring size 70+70. I need to add more capacity
to meet production demand, but when I add 71st node, then it streams data
from other nodes as expected, but after some-time it spends an enormous
amount of time doing GCs, and freezes.
Snippet from the log file...
{"@timestamp":"2022-04-23T02:21:38.030+00:00","@version":1,"message":"G1
Old Generation GC in 18288ms. G1 Old Gen: 34206148032 -> 33725019032;
","logger_name":"o.a.c.service.GCInspector","thread_name":"Service
Thread","level":"WARN","level_value":30000}
The new node has around 1.5M SSTables (from multiple tables). If I reduce
the SSTable count to below 500K, then it joins fine.
I am using LCS compaction with default settings. I've tried changing to
STCS, but no luck :(
Any help would be highly appreciated. Thanks a lot!
Jaydeep
