Have you tried this? https://github.com/apache/drill/commit/48b3fd2ee040f30be3ff3172bf03111ca58d4b48
On Thu, Aug 11, 2016 at 11:13:10AM +0000, Avi Haleva wrote:
Embedding the thread dump content in the message : Thread dump 1: 2016-08-11 11:02:19 Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode): "Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800 nid=0x1cce6 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800 nid=0x2490 runnable [0x00007f4c000b6000] java.lang.Thread.State: RUNNABLE at java.lang.UNIXProcess.waitForProcessExit(Native Method) at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:290) at java.lang.UNIXProcess$$Lambda$7/2030562336.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000006cdb44650> (a java.util.concurrent.ThreadPoolExecutor$Worker) "threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800 nid=0x1dc04 waiting on condition [0x00007f4bed9d6000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03 runnable [0x00007f4bedad7000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800 nid=0x1dc01 waiting on condition [0x00007f4c005c1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c0013c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000 nid=0x1dc00 waiting on condition [0x00007f4c006c2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c0023678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at java.util.concurrent.DelayQueue.take(DelayQueue.java:70) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000006c00237b8> (a java.util.concurrent.ThreadPoolExecutor$Worker) "main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800 nid=0x1dbff waiting on condition [0x00007f4c007c3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c001bff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Locked ownable synchronizers: - None "main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0 tid=0x00007f4c4d420000 nid=0x1dbfe runnable [0x00007f4c008c4000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000006c002b708> (a sun.nio.ch.Util$2) - locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet) - locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Locked ownable synchronizers: - None "Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0 tid=0x00007f4c4d32c800 nid=0x1dbfd waiting on condition [0x00007f4c009c5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c0033630> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245) at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43) at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000006c0033750> (a java.util.concurrent.ThreadPoolExecutor$Worker) "NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0 tid=0x00007f4c4cdab800 nid=0x1dbfc in Object.wait() [0x00007f4c01b52000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream) at jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) - locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 nid=0x1dbe9 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000 nid=0x1dbe8 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000 nid=0x1dbe7 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800 nid=0x1dbe6 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800 nid=0x1dbe5 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000 nid=0x1dbe4 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000 nid=0x1dbe3 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000 nid=0x1dbe2 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000 nid=0x1dbe1 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800 nid=0x1dbe0 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800 nid=0x1dbdf waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800 nid=0x1dbde waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000 nid=0x1dbdd waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000 nid=0x1dbdc runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in Object.wait() [0x00007f4c036f5000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) - locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Locked ownable synchronizers: - None "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000 nid=0x1dbda in Object.wait() [0x00007f4c037f6000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Locked ownable synchronizers: - None "main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb runnable [0x00007f4c551a8000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x000000078ed7ef50> (a java.lang.UNIXProcess$ProcessPipeInputStream) at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:191) at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173) at jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168) at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76) at jline.UnixTerminal.disableInterruptCharacter(UnixTerminal.java:109) at jline.console.ConsoleReader.readLine(ConsoleReader.java:2194) at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126) at sqlline.SqlLine.begin(SqlLine.java:621) at sqlline.SqlLine.start(SqlLine.java:375) at sqlline.SqlLine.main(SqlLine.java:268) Locked ownable synchronizers: - None "VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 nid=0x1dbcc runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 nid=0x1dbcd runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 nid=0x1dbce runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 nid=0x1dbcf runnable "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 nid=0x1dbd0 runnable "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 nid=0x1dbd1 runnable "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 nid=0x1dbd2 runnable "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 nid=0x1dbd3 runnable "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 nid=0x1dbd4 runnable "GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 nid=0x1dbd5 runnable "GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 nid=0x1dbd6 runnable "GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 nid=0x1dbd7 runnable "GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 nid=0x1dbd8 runnable "VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea waiting on condition JNI global references: 314 Thread dump 2: 2016-08-11 11:02:22 Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode): "Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800 nid=0x1cce6 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800 nid=0x2490 runnable [0x00007f4c000b6000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.available(Native Method) at java.lang.UNIXProcess$ProcessPipeInputStream.drainInputStream(UNIXProcess.java:519) at java.lang.UNIXProcess$ProcessPipeInputStream.processExited(UNIXProcess.java:533) - locked <0x0000000792eb6e40> (a java.lang.Object) - locked <0x0000000792eb4d90> (a java.lang.UNIXProcess$ProcessPipeInputStream) at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:299) at java.lang.UNIXProcess$$Lambda$7/2030562336.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000006cdb44650> (a java.util.concurrent.ThreadPoolExecutor$Worker) "threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800 nid=0x1dc04 waiting on condition [0x00007f4bed9d6000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03 runnable [0x00007f4bedad7000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800 nid=0x1dc01 waiting on condition [0x00007f4c005c1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c0013c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000 nid=0x1dc00 waiting on condition [0x00007f4c006c2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c0023678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at java.util.concurrent.DelayQueue.take(DelayQueue.java:70) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000006c00237b8> (a java.util.concurrent.ThreadPoolExecutor$Worker) "main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800 nid=0x1dbff waiting on condition [0x00007f4c007c3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c001bff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Locked ownable synchronizers: - None "main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0 tid=0x00007f4c4d420000 nid=0x1dbfe runnable [0x00007f4c008c4000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000006c002b708> (a sun.nio.ch.Util$2) - locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet) - locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Locked ownable synchronizers: - None "Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0 tid=0x00007f4c4d32c800 nid=0x1dbfd waiting on condition [0x00007f4c009c5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c0033630> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245) at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43) at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000006c0033750> (a java.util.concurrent.ThreadPoolExecutor$Worker) "NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0 tid=0x00007f4c4cdab800 nid=0x1dbfc in Object.wait() [0x00007f4c01b52000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream) at jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) - locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 nid=0x1dbe9 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000 nid=0x1dbe8 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000 nid=0x1dbe7 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800 nid=0x1dbe6 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800 nid=0x1dbe5 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000 nid=0x1dbe4 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000 nid=0x1dbe3 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000 nid=0x1dbe2 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000 nid=0x1dbe1 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800 nid=0x1dbe0 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800 nid=0x1dbdf waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800 nid=0x1dbde waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000 nid=0x1dbdd waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000 nid=0x1dbdc runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in Object.wait() [0x00007f4c036f5000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) - locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Locked ownable synchronizers: - None "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000 nid=0x1dbda in Object.wait() [0x00007f4c037f6000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Locked ownable synchronizers: - None "main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb in Object.wait() [0x00007f4c551a8000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396) - locked <0x0000000792eb2c68> (a java.lang.UNIXProcess) at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:199) at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173) at jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168) at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76) at jline.UnixTerminal.enableInterruptCharacter(UnixTerminal.java:122) at jline.console.ConsoleReader.readLine(ConsoleReader.java:2859) at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126) at sqlline.SqlLine.begin(SqlLine.java:621) at sqlline.SqlLine.start(SqlLine.java:375) at sqlline.SqlLine.main(SqlLine.java:268) Locked ownable synchronizers: - None "VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 nid=0x1dbcc runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 nid=0x1dbcd runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 nid=0x1dbce runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 nid=0x1dbcf runnable "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 nid=0x1dbd0 runnable "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 nid=0x1dbd1 runnable "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 nid=0x1dbd2 runnable "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 nid=0x1dbd3 runnable "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 nid=0x1dbd4 runnable "GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 nid=0x1dbd5 runnable "GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 nid=0x1dbd6 runnable "GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 nid=0x1dbd7 runnable "GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 nid=0x1dbd8 runnable "VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea waiting on condition JNI global references: 314 Thread dump 3: 2016-08-11 11:02:25 Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode): "Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800 nid=0x1cce6 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800 nid=0x2490 waiting on condition [0x00007f4c000b6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c00435c8> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800 nid=0x1dc04 waiting on condition [0x00007f4bed9d6000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03 runnable [0x00007f4bedad7000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:148) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800 nid=0x1dc01 waiting on condition [0x00007f4c005c1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c0013c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000 nid=0x1dc00 waiting on condition [0x00007f4c006c2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c0023678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at java.util.concurrent.DelayQueue.take(DelayQueue.java:70) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000006c00237b8> (a java.util.concurrent.ThreadPoolExecutor$Worker) "main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800 nid=0x1dbff waiting on condition [0x00007f4c007c3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c001bff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Locked ownable synchronizers: - None "main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0 tid=0x00007f4c4d420000 nid=0x1dbfe runnable [0x00007f4c008c4000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000006c002b708> (a sun.nio.ch.Util$2) - locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet) - locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Locked ownable synchronizers: - None "Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0 tid=0x00007f4c4d32c800 nid=0x1dbfd waiting on condition [0x00007f4c009c5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c0033630> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245) at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43) at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000006c0033750> (a java.util.concurrent.ThreadPoolExecutor$Worker) "NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0 tid=0x00007f4c4cdab800 nid=0x1dbfc in Object.wait() [0x00007f4c01b52000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream) at jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) - locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 nid=0x1dbe9 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000 nid=0x1dbe8 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000 nid=0x1dbe7 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800 nid=0x1dbe6 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800 nid=0x1dbe5 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000 nid=0x1dbe4 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000 nid=0x1dbe3 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000 nid=0x1dbe2 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000 nid=0x1dbe1 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800 nid=0x1dbe0 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800 nid=0x1dbdf waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800 nid=0x1dbde waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000 nid=0x1dbdd waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000 nid=0x1dbdc runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in Object.wait() [0x00007f4c036f5000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) - locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Locked ownable synchronizers: - None "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000 nid=0x1dbda in Object.wait() [0x00007f4c037f6000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Locked ownable synchronizers: - None "main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb runnable [0x00007f4c551a8000] java.lang.Thread.State: RUNNABLE at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) at java.lang.Runtime.exec(Runtime.java:620) at java.lang.Runtime.exec(Runtime.java:485) at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:183) at jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173) at jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168) at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76) at jline.UnixTerminal.enableInterruptCharacter(UnixTerminal.java:122) at jline.console.ConsoleReader.readLine(ConsoleReader.java:2859) at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126) at sqlline.SqlLine.begin(SqlLine.java:621) at sqlline.SqlLine.start(SqlLine.java:375) at sqlline.SqlLine.main(SqlLine.java:268) Locked ownable synchronizers: - None "VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 nid=0x1dbcc runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 nid=0x1dbcd runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 nid=0x1dbce runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 nid=0x1dbcf runnable "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 nid=0x1dbd0 runnable "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 nid=0x1dbd1 runnable "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 nid=0x1dbd2 runnable "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 nid=0x1dbd3 runnable "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 nid=0x1dbd4 runnable "GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 nid=0x1dbd5 runnable "GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 nid=0x1dbd6 runnable "GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 nid=0x1dbd7 runnable "GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 nid=0x1dbd8 runnable "VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea waiting on condition JNI global references: 314 -----Original Message----- From: Avi Haleva [mailto:[email protected]] Sent: Thursday, August 11, 2016 14:10 To: [email protected] Subject: RE: Drill CPU Usage Both the log and out files looks clean Attached you can find the thread dump output (3 snapshots). I will appreciate if one can point me to the reason the drillbits consume constantly CPU BTW, the CPU usage is now a constant at 40% for a single core (double from a week ago) -----Original Message----- From: Dechang Gu [mailto:[email protected]] Sent: Tuesday, August 2, 2016 22:21 To: [email protected] Subject: Re: Drill CPU Usage On Tue, Aug 2, 2016 at 12:49 AM, Avi Haleva <[email protected]> wrote:Hi, I'm in the process of evaluating Drill as a analytic repositiory. I've noticed that even when idle, the drillbit process consume 20% of a single core constantly. Is that expected behavior or have I miss configured something.I don't think that is expected. I checked my system, the drillbit CPU usage stays at 0% almost constantly when idle. I did see occasionally it has a glitch at 20-30% CPU usage, due to zookeeper heartbeat traffic, I guess. Can you collect jstack on the drillbit process to find out what is running? Also check drillbit.out and drillbit.log (typically under /var/log/drill/ ) to see if there is anything suspicious. HTH, and Thanks, DechangI'm running drill in single node with zookeeper in a non-embeded mode (as I want to have more than a single connection concurrently) Thanks, Avi
--
Oscar Morante
"Self-education is, I firmly believe, the only kind of education there is."
-- Isaac Asimov.
signature.asc
Description: Digital signature
