I have limited options to use JDK based tools because in our environment we are running JRE .
I tried to debug more and could see using top that Command is MutationStage in top output , Any clue we get from this ? top - 16:30:47 up 94 days, 5:33, 1 user, load average: 134.83, 142.48, 144.75 Tasks: 564 total, 58 running, 506 sleeping, 0 stopped, 0 zombie Cpu(s): 95.2%us, 2.5%sy, 0.3%ni, 1.7%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 132236016k total, 131378384k used, 857632k free, 189208k buffers Swap: 0k total, 0k used, 0k free, 94530140k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ WCHAN COMMAND 11798 cassandr 20 0 261g 42g 14g R 14.4 33.3 76:47.38 - MutationStage-1 11762 cassandr 20 0 261g 42g 14g S 14.1 33.3 82:19.84 - MutationStage-9 11530 cassandr 20 0 261g 42g 14g R 13.8 33.3 100:00.22 - MutationStage-3 11501 cassandr 20 0 261g 42g 14g S 13.4 33.3 2598:38 - MutationStage-5 11688 cassandr 20 0 261g 42g 14g S 13.1 33.3 90:42.47 - MutationStage-5 11512 cassandr 20 0 261g 42g 14g R 12.8 33.3 153:13.59 - MutationStage-1 11534 cassandr 20 0 261g 42g 14g R 12.8 33.3 104:48.21 - MutationStage-2 11708 cassandr 20 0 261g 42g 14g S 12.5 33.3 87:17.64 - MutationStage-6 11783 cassandr 20 0 261g 42g 14g S 12.5 33.3 76:01.10 futex_wai MutationStage-1 11792 cassandr 20 0 261g 42g 14g S 12.5 33.3 76:19.90 futex_wai MutationStage-1 11504 cassandr 20 0 261g 42g 14g S 12.2 33.3 859:10.54 futex_wai MutationStage-8 11517 cassandr 20 0 261g 42g 14g R 12.2 33.3 116:18.38 - MutationStage-2 11535 cassandr 20 0 261g 42g 14g R 12.2 33.3 96:11.11 - MutationStage-3 11710 cassandr 20 0 261g 42g 14g R 12.2 33.3 86:50.77 - MutationStage-7 11730 cassandr 20 0 261g 42g 14g S 12.2 33.3 78:36.04 - MutationStage-1 11743 cassandr 20 0 261g 42g 14g R 12.2 33.3 80:27.18 - MutationStage-1 11773 cassandr 20 0 261g 42g 14g R 12.2 33.3 79:29.48 - MutationStage-1 11800 cassandr 20 0 261g 42g 14g S 12.2 33.3 77:01.39 futex_wai MutationStage-1 11830 cassandr 20 0 261g 42g 14g R 12.2 33.3 70:47.18 - MutationStage-1 11495 cassandr 20 0 261g 42g 14g R 11.8 33.3 7693:04 - MutationStage-3 11675 cassandr 20 0 261g 42g 14g R 11.8 33.3 94:13.22 - MutationStage-4 11683 cassandr 20 0 261g 42g 14g S 11.8 33.3 91:42.91 futex_wai MutationStage-4 11701 cassandr 20 0 261g 42g 14g S 11.8 33.3 85:16.00 - MutationStage-7 11703 cassandr 20 0 261g 42g 14g R 11.8 33.3 88:33.81 - MutationStage-6 11725 cassandr 20 0 261g 42g 14g R 11.8 33.3 78:12.70 - MutationStage-1 11752 cassandr 20 0 261g 42g 14g S 11.8 33.3 83:25.14 futex_wai MutationStage-9 11755 cassandr 20 0 261g 42g 14g R 11.8 33.3 82:38.87 - MutationStage-9 11776 cassandr 20 0 261g 42g 14g S 11.8 33.3 79:31.49 futex_wai MutationStage-1 11781 cassandr 20 0 261g 42g 14g R 11.8 33.3 75:01.54 - MutationStage-1 11796 cassandr 20 0 261g 42g 14g S 11.8 33.3 77:03.78 - MutationStage-1 11804 cassandr 20 0 261g 42g 14g R 11.8 33.3 81:38.46 - MutationStage-1 11818 cassandr 20 0 261g 42g 14g S 11.8 33.3 76:51.42 - MutationStage-1 11823 cassandr 20 0 261g 42g 14g R 11.8 33.3 75:56.69 - MutationStage-1 11506 cassandr 20 0 261g 42g 14g R 11.5 33.3 502:50.67 - MutationStage-1 11513 cassandr 20 0 261g 42g 14g R 11.5 33.3 140:00.60 - MutationStage-1 11515 cassandr 20 0 261g 42g 14g S 11.5 33.3 123:31.16 futex_wai MutationStage-1 11676 cassandr 20 0 261g 42g 14g S 11.5 33.3 93:44.36 futex_wai MutationStage-4 11680 cassandr 20 0 261g 42g 14g S 11.5 33.3 93:28.55 futex_wai MutationStage-4 11706 cassandr 20 0 261g 42g 14g R 11.5 33.3 89:17.10 - MutationStage-6 11729 cassandr 20 0 261g 42g 14g R 11.5 33.3 78:42.33 - MutationStage-1 On Thu, 6 Feb 2020 at 10:17, Elliott Sims <elli...@backblaze.com> wrote: > Async-profiler (https://github.com/jvm-profiling-tools/async-profiler ) > flamegraphs can also be a really good tool to figure out the exact > callgraph that's leading to the futex_wait, both in and out of the JVM. >
