Re: Soon After Starting c* Process: CPU 100% for java Process

2021-07-02 Thread Bowen Song
On 01/07/2021 23:41, Elliott Sims wrote: Also for narrowing down performance issues, I've had good luck with the "ttop" module of Swiss Java Knife and with the async-profiler tool: https://github.com/jvm-profiling-tools/async-profiler

Re: Soon After Starting c* Process: CPU 100% for java Process

2021-07-02 Thread Kane Wilson
> > Eventually, the shared pool worker crashes > -- > b-3223-big-Data.db > WARN [SharedPool-Worker-55] 2021-06-30 19:55:41,677 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread >

Re: Soon After Starting c* Process: CPU 100% for java Process

2021-07-01 Thread Elliott Sims
As more general advice, I'd strongly encourage you to update to 3.11.x from 2.2.8. My personal experience is that it's significantly faster and more space-efficient, and the garbage collection behavior under pressure is drastically better. There's also improved tooling for diagnosing performance

Re: Soon After Starting c* Process: CPU 100% for java Process

2021-07-01 Thread Fred Habash
Great. Thanks. I examined the debug logs and from the time c* starts till it crashes the 30 minute duration log spitting the exact same message. Google research leads to no clear answers as to ... - What is 'evicting cold readers' mean? - Why is evicting the same sstable throughout i.e. from

Re: Soon After Starting c* Process: CPU 100% for java Process

2021-06-30 Thread Kane Wilson
Looks like it's doing a lot of reads immediately on startup (AbstractQueryPager) which is potentially causing a lot of GC (guessing that's what caused the StatusLogger). DEBUG [SharedPool-Worker-113] 2021-06-30 13:39:04,766 AbstractQueryPager.java:133 - Remaining rows to page: 2147483646 is

Soon After Starting c* Process: CPU 100% for java Process

2021-06-30 Thread Fred Habash
I have node in cluster when I start c, the cpu reaches 100% with java process on top. Within a few minutes, jvm crashes (jvm instability) messages in system.log and c* crashes. Once c* is up, cluster average read latency reaches multi-seconds and client apps are unhappy. For now, the only way out