Embedding the thread dump content in the message :
Thread dump 1:
2016-08-11 11:02:19
Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):
"Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800
nid=0x1cce6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800
nid=0x2490 runnable [0x00007f4c000b6000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.waitForProcessExit(Native Method)
at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:290)
at java.lang.UNIXProcess$$Lambda$7/2030562336.run(Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006cdb44650> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800
nid=0x1dc04 waiting on condition [0x00007f4bed9d6000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03
runnable [0x00007f4bedad7000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:148)
at
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800
nid=0x1dc01 waiting on condition [0x00007f4c005c1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0013c68> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000
nid=0x1dc00 waiting on condition [0x00007f4c006c2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0023678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006c00237b8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800
nid=0x1dbff waiting on condition [0x00007f4c007c3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c001bff8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Locked ownable synchronizers:
- None
"main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0
tid=0x00007f4c4d420000 nid=0x1dbfe runnable [0x00007f4c008c4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c002b708> (a sun.nio.ch.Util$2)
- locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Locked ownable synchronizers:
- None
"Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0
tid=0x00007f4c4d32c800 nid=0x1dbfd waiting on condition [0x00007f4c009c5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0033630> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
at
org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
at
org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006c0033750> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0
tid=0x00007f4c4cdab800 nid=0x1dbfc in Object.wait() [0x00007f4c01b52000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c003b5f0> (a
jline.internal.NonBlockingInputStream)
at
jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278)
- locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 nid=0x1dbe9
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000
nid=0x1dbe8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000
nid=0x1dbe7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800
nid=0x1dbe6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800
nid=0x1dbe5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000
nid=0x1dbe4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000
nid=0x1dbe3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000
nid=0x1dbe2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000
nid=0x1dbe1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800
nid=0x1dbe0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800
nid=0x1dbdf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800
nid=0x1dbde waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000
nid=0x1dbdd waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000
nid=0x1dbdc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in
Object.wait() [0x00007f4c036f5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000
nid=0x1dbda in Object.wait() [0x00007f4c037f6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb runnable
[0x00007f4c551a8000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x000000078ed7ef50> (a
java.lang.UNIXProcess$ProcessPipeInputStream)
at
jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:191)
at
jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173)
at
jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168)
at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76)
at jline.UnixTerminal.disableInterruptCharacter(UnixTerminal.java:109)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2194)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126)
at sqlline.SqlLine.begin(SqlLine.java:621)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 nid=0x1dbcc
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 nid=0x1dbcd
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 nid=0x1dbce
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 nid=0x1dbcf
runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 nid=0x1dbd0
runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 nid=0x1dbd1
runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 nid=0x1dbd2
runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 nid=0x1dbd3
runnable
"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 nid=0x1dbd4
runnable
"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 nid=0x1dbd5
runnable
"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 nid=0x1dbd6
runnable
"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 nid=0x1dbd7
runnable
"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 nid=0x1dbd8
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea waiting
on condition
JNI global references: 314
Thread dump 2:
2016-08-11 11:02:22
Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):
"Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800
nid=0x1cce6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800
nid=0x2490 runnable [0x00007f4c000b6000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.available(Native Method)
at
java.lang.UNIXProcess$ProcessPipeInputStream.drainInputStream(UNIXProcess.java:519)
at
java.lang.UNIXProcess$ProcessPipeInputStream.processExited(UNIXProcess.java:533)
- locked <0x0000000792eb6e40> (a java.lang.Object)
- locked <0x0000000792eb4d90> (a
java.lang.UNIXProcess$ProcessPipeInputStream)
at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:299)
at java.lang.UNIXProcess$$Lambda$7/2030562336.run(Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006cdb44650> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800
nid=0x1dc04 waiting on condition [0x00007f4bed9d6000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03
runnable [0x00007f4bedad7000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:148)
at
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800
nid=0x1dc01 waiting on condition [0x00007f4c005c1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0013c68> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000
nid=0x1dc00 waiting on condition [0x00007f4c006c2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0023678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006c00237b8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800
nid=0x1dbff waiting on condition [0x00007f4c007c3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c001bff8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Locked ownable synchronizers:
- None
"main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0
tid=0x00007f4c4d420000 nid=0x1dbfe runnable [0x00007f4c008c4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c002b708> (a sun.nio.ch.Util$2)
- locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Locked ownable synchronizers:
- None
"Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0
tid=0x00007f4c4d32c800 nid=0x1dbfd waiting on condition [0x00007f4c009c5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0033630> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
at
org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
at
org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006c0033750> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0
tid=0x00007f4c4cdab800 nid=0x1dbfc in Object.wait() [0x00007f4c01b52000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c003b5f0> (a
jline.internal.NonBlockingInputStream)
at
jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278)
- locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 nid=0x1dbe9
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000
nid=0x1dbe8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000
nid=0x1dbe7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800
nid=0x1dbe6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800
nid=0x1dbe5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000
nid=0x1dbe4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000
nid=0x1dbe3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000
nid=0x1dbe2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000
nid=0x1dbe1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800
nid=0x1dbe0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800
nid=0x1dbdf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800
nid=0x1dbde waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000
nid=0x1dbdd waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000
nid=0x1dbdc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in
Object.wait() [0x00007f4c036f5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000
nid=0x1dbda in Object.wait() [0x00007f4c037f6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb in Object.wait()
[0x00007f4c551a8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
- locked <0x0000000792eb2c68> (a java.lang.UNIXProcess)
at
jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:199)
at
jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173)
at
jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168)
at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76)
at jline.UnixTerminal.enableInterruptCharacter(UnixTerminal.java:122)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2859)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126)
at sqlline.SqlLine.begin(SqlLine.java:621)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 nid=0x1dbcc
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 nid=0x1dbcd
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 nid=0x1dbce
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 nid=0x1dbcf
runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 nid=0x1dbd0
runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 nid=0x1dbd1
runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 nid=0x1dbd2
runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 nid=0x1dbd3
runnable
"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 nid=0x1dbd4
runnable
"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 nid=0x1dbd5
runnable
"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 nid=0x1dbd6
runnable
"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 nid=0x1dbd7
runnable
"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 nid=0x1dbd8
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea waiting
on condition
JNI global references: 314
Thread dump 3:
2016-08-11 11:02:25
Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode):
"Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800
nid=0x1cce6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800
nid=0x2490 waiting on condition [0x00007f4c000b6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c00435c8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800
nid=0x1dc04 waiting on condition [0x00007f4bed9d6000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03
runnable [0x00007f4bedad7000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:148)
at
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800
nid=0x1dc01 waiting on condition [0x00007f4c005c1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0013c68> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000
nid=0x1dc00 waiting on condition [0x00007f4c006c2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0023678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
at
org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006c00237b8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800
nid=0x1dbff waiting on condition [0x00007f4c007c3000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c001bff8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Locked ownable synchronizers:
- None
"main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0
tid=0x00007f4c4d420000 nid=0x1dbfe runnable [0x00007f4c008c4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c002b708> (a sun.nio.ch.Util$2)
- locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Locked ownable synchronizers:
- None
"Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0
tid=0x00007f4c4d32c800 nid=0x1dbfd waiting on condition [0x00007f4c009c5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0033630> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
at
org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
at
org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006c0033750> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0
tid=0x00007f4c4cdab800 nid=0x1dbfc in Object.wait() [0x00007f4c01b52000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c003b5f0> (a
jline.internal.NonBlockingInputStream)
at
jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278)
- locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 nid=0x1dbe9
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000
nid=0x1dbe8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000
nid=0x1dbe7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800
nid=0x1dbe6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800
nid=0x1dbe5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000
nid=0x1dbe4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000
nid=0x1dbe3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000
nid=0x1dbe2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000
nid=0x1dbe1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800
nid=0x1dbe0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800
nid=0x1dbdf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800
nid=0x1dbde waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000
nid=0x1dbdd waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000
nid=0x1dbdc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in
Object.wait() [0x00007f4c036f5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000
nid=0x1dbda in Object.wait() [0x00007f4c037f6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb runnable
[0x00007f4c551a8000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
at java.lang.Runtime.exec(Runtime.java:620)
at java.lang.Runtime.exec(Runtime.java:485)
at
jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:183)
at
jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173)
at
jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168)
at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76)
at jline.UnixTerminal.enableInterruptCharacter(UnixTerminal.java:122)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2859)
at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126)
at sqlline.SqlLine.begin(SqlLine.java:621)
at sqlline.SqlLine.start(SqlLine.java:375)
at sqlline.SqlLine.main(SqlLine.java:268)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 nid=0x1dbcc
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 nid=0x1dbcd
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 nid=0x1dbce
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 nid=0x1dbcf
runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 nid=0x1dbd0
runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 nid=0x1dbd1
runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 nid=0x1dbd2
runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 nid=0x1dbd3
runnable
"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 nid=0x1dbd4
runnable
"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 nid=0x1dbd5
runnable
"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 nid=0x1dbd6
runnable
"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 nid=0x1dbd7
runnable
"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 nid=0x1dbd8
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea waiting
on condition
JNI global references: 314
-----Original Message-----
From: Avi Haleva [mailto:[email protected]]
Sent: Thursday, August 11, 2016 14:10
To: [email protected]
Subject: RE: Drill CPU Usage
Both the log and out files looks clean
Attached you can find the thread dump output (3 snapshots).
I will appreciate if one can point me to the reason the drillbits consume
constantly CPU
BTW, the CPU usage is now a constant at 40% for a single core (double from a
week ago)
-----Original Message-----
From: Dechang Gu [mailto:[email protected]]
Sent: Tuesday, August 2, 2016 22:21
To: [email protected]
Subject: Re: Drill CPU Usage
On Tue, Aug 2, 2016 at 12:49 AM, Avi Haleva <[email protected]> wrote:
> Hi,
> I'm in the process of evaluating Drill as a analytic repositiory.
> I've noticed that even when idle, the drillbit process consume 20% of
> a single core constantly.
>
> Is that expected behavior or have I miss configured something.
>
I don't think that is expected. I checked my system, the drillbit CPU usage
stays at 0% almost constantly when idle.
I did see occasionally it has a glitch at 20-30% CPU usage, due to zookeeper
heartbeat traffic, I guess.
Can you collect jstack on the drillbit process to find out what is running?
Also check drillbit.out and drillbit.log (typically under /var/log/drill/ ) to
see if there is anything suspicious.
HTH, and Thanks,
Dechang
>
> I'm running drill in single node with zookeeper in a non-embeded mode
> (as I want to have more than a single connection concurrently)
>
> Thanks,
> Avi
>
>
