We have 5 servers in the ensemble with a LeaderSelector in each server. In our performance environment, we are noticing that the server that takes leadership causes CPU utilization to run up. Test yesterday say CPU utilization at 95%. Today was better at 33%. The odd thing is that even after the test are completed, the CPU utilization does not go down.
We determined what thread was causing the high CPU utilization with ps -C java -L -o pid,tid,pcpu,state,stime,time,nlwp The TID is the thread id. We then converted the TID to hex and searched the javacore thread dump to see what was causing the CPU to hang. Every time the consuming thread was running the LeaderSelector. 3XMTHREADINFO "LeaderSelector-2" J9VMThread:0x00000000C54D8800, j9thread_t:0x00007F8C51E5CC80, java/lang/Thread:0x0000000006FAB4D8, state:CW, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x3A8, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x2BDE, native priority:0x5, native policy:UNKNOWN) 3XMTHREADINFO2 (native stack address range from:0x00007F8BD298F000, to:0x00007F8BD29D0000, size:0x41000) 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at com/lps/mg/inlet/sadevents/Leader.takeLeadership(Leader.java:127(Compiled Code)) 4XESTACKTRACE at org/apache/curator/framework/recipes/leader/LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536) 4XESTACKTRACE at org/apache/curator/framework/recipes/leader/LeaderSelector.doWork(LeaderSelector.java:391) 4XESTACKTRACE at org/apache/curator/framework/recipes/leader/LeaderSelector.doWorkLoop(LeaderSelector.java:436) 4XESTACKTRACE at org/apache/curator/framework/recipes/leader/LeaderSelector.access$100(LeaderSelector.java:64) 4XESTACKTRACE at org/apache/curator/framework/recipes/leader/LeaderSelector$2.call(LeaderSelector.java:241) 4XESTACKTRACE at org/apache/curator/framework/recipes/leader/LeaderSelector$2.call(LeaderSelector.java:235) 4XESTACKTRACE at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314) 4XESTACKTRACE at java/util/concurrent/FutureTask.run(FutureTask.java:149) 4XESTACKTRACE at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:450) 4XESTACKTRACE at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314) 4XESTACKTRACE at java/util/concurrent/FutureTask.run(FutureTask.java:149) 4XESTACKTRACE at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908) 4XESTACKTRACE at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931) 4XESTACKTRACE at java/lang/Thread.run(Thread.java:773(Compiled Code)) Is this a reason that the LeaderSelector would run away with the CPU? Thank you, Curtis Cantrell The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
