I'm using version 1.7.0 , I guess I need to apply to patch to test this fix. I haven't seen any reference to CPU usage in the bug report, am I missing something ?
-----Original Message----- From: Oscar Morante [mailto:[email protected]] Sent: Monday, August 15, 2016 13:54 To: [email protected] Subject: Re: Drill CPU Usage Have you tried this? https://github.com/apache/drill/commit/48b3fd2ee040f30be3ff3172bf03111ca58d4b48 On Thu, Aug 11, 2016 at 11:13:10AM +0000, Avi Haleva wrote: >Embedding the thread dump content in the message : > >Thread dump 1: > >2016-08-11 11:02:19 >Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode): > >"Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800 >nid=0x1cce6 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800 >nid=0x2490 runnable [0x00007f4c000b6000] > java.lang.Thread.State: RUNNABLE > at java.lang.UNIXProcess.waitForProcessExit(Native Method) > at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:290) > at java.lang.UNIXProcess$$Lambda$7/2030562336.run(Unknown Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - <0x00000006cdb44650> (a >java.util.concurrent.ThreadPoolExecutor$Worker) > >"threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800 >nid=0x1dc04 waiting on condition [0x00007f4bed9d6000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03 >runnable [0x00007f4bedad7000] > java.lang.Thread.State: RUNNABLE > at io.netty.channel.epoll.Native.epollWait0(Native Method) > at io.netty.channel.epoll.Native.epollWait(Native.java:148) > at > io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180) > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800 >nid=0x1dc01 waiting on condition [0x00007f4c005c1000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c0013c68> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000 >nid=0x1dc00 waiting on condition [0x00007f4c006c2000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c0023678> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:70) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - <0x00000006c00237b8> (a >java.util.concurrent.ThreadPoolExecutor$Worker) > >"main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800 >nid=0x1dbff waiting on condition [0x00007f4c007c3000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c001bff8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at >org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) > > Locked ownable synchronizers: > - None > >"main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0 >tid=0x00007f4c4d420000 nid=0x1dbfe runnable [0x00007f4c008c4000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000006c002b708> (a sun.nio.ch.Util$2) > - locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) > at >org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) > > Locked ownable synchronizers: > - None > >"Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0 >tid=0x00007f4c4d32c800 nid=0x1dbfd waiting on condition [0x00007f4c009c5000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c0033630> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) > at > org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245) > at > org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43) > at > org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - <0x00000006c0033750> (a >java.util.concurrent.ThreadPoolExecutor$Worker) > >"NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0 >tid=0x00007f4c4cdab800 nid=0x1dbfc in Object.wait() [0x00007f4c01b52000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000006c003b5f0> (a > jline.internal.NonBlockingInputStream) > at > jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) > - locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 >nid=0x1dbe9 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000 >nid=0x1dbe8 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000 >nid=0x1dbe7 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800 >nid=0x1dbe6 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800 >nid=0x1dbe5 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000 >nid=0x1dbe4 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000 >nid=0x1dbe3 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000 >nid=0x1dbe2 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000 >nid=0x1dbe1 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800 >nid=0x1dbe0 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800 >nid=0x1dbdf waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800 >nid=0x1dbde waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000 >nid=0x1dbdd waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000 >nid=0x1dbdc runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in >Object.wait() [0x00007f4c036f5000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > - locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > > Locked ownable synchronizers: > - None > >"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000 >nid=0x1dbda in Object.wait() [0x00007f4c037f6000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > - locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > > Locked ownable synchronizers: > - None > >"main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb runnable >[0x00007f4c551a8000] > java.lang.Thread.State: RUNNABLE > at java.io.FileInputStream.readBytes(Native Method) > at java.io.FileInputStream.read(FileInputStream.java:255) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read(BufferedInputStream.java:265) > - locked <0x000000078ed7ef50> (a > java.lang.UNIXProcess$ProcessPipeInputStream) > at > jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:191) > at > jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173) > at > jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168) > at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76) > at jline.UnixTerminal.disableInterruptCharacter(UnixTerminal.java:109) > at jline.console.ConsoleReader.readLine(ConsoleReader.java:2194) > at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126) > at sqlline.SqlLine.begin(SqlLine.java:621) > at sqlline.SqlLine.start(SqlLine.java:375) > at sqlline.SqlLine.main(SqlLine.java:268) > > Locked ownable synchronizers: > - None > >"VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable > >"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 >nid=0x1dbcc runnable > >"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 >nid=0x1dbcd runnable > >"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 >nid=0x1dbce runnable > >"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 >nid=0x1dbcf runnable > >"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 >nid=0x1dbd0 runnable > >"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 >nid=0x1dbd1 runnable > >"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 >nid=0x1dbd2 runnable > >"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 >nid=0x1dbd3 runnable > >"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 >nid=0x1dbd4 runnable > >"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 >nid=0x1dbd5 runnable > >"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 >nid=0x1dbd6 runnable > >"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 >nid=0x1dbd7 runnable > >"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 >nid=0x1dbd8 runnable > >"VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea >waiting on condition > >JNI global references: 314 > > >Thread dump 2: >2016-08-11 11:02:22 >Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode): > >"Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800 >nid=0x1cce6 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800 >nid=0x2490 runnable [0x00007f4c000b6000] > java.lang.Thread.State: RUNNABLE > at java.io.FileInputStream.available(Native Method) > at > java.lang.UNIXProcess$ProcessPipeInputStream.drainInputStream(UNIXProcess.java:519) > at > java.lang.UNIXProcess$ProcessPipeInputStream.processExited(UNIXProcess.java:533) > - locked <0x0000000792eb6e40> (a java.lang.Object) > - locked <0x0000000792eb4d90> (a > java.lang.UNIXProcess$ProcessPipeInputStream) > at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:299) > at java.lang.UNIXProcess$$Lambda$7/2030562336.run(Unknown Source) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - <0x00000006cdb44650> (a >java.util.concurrent.ThreadPoolExecutor$Worker) > >"threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800 >nid=0x1dc04 waiting on condition [0x00007f4bed9d6000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03 >runnable [0x00007f4bedad7000] > java.lang.Thread.State: RUNNABLE > at io.netty.channel.epoll.Native.epollWait0(Native Method) > at io.netty.channel.epoll.Native.epollWait(Native.java:148) > at > io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180) > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800 >nid=0x1dc01 waiting on condition [0x00007f4c005c1000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c0013c68> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000 >nid=0x1dc00 waiting on condition [0x00007f4c006c2000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c0023678> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:70) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - <0x00000006c00237b8> (a >java.util.concurrent.ThreadPoolExecutor$Worker) > >"main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800 >nid=0x1dbff waiting on condition [0x00007f4c007c3000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c001bff8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at >org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) > > Locked ownable synchronizers: > - None > >"main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0 >tid=0x00007f4c4d420000 nid=0x1dbfe runnable [0x00007f4c008c4000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000006c002b708> (a sun.nio.ch.Util$2) > - locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) > at >org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) > > Locked ownable synchronizers: > - None > >"Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0 >tid=0x00007f4c4d32c800 nid=0x1dbfd waiting on condition [0x00007f4c009c5000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c0033630> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) > at > org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245) > at > org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43) > at > org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - <0x00000006c0033750> (a >java.util.concurrent.ThreadPoolExecutor$Worker) > >"NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0 >tid=0x00007f4c4cdab800 nid=0x1dbfc in Object.wait() [0x00007f4c01b52000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000006c003b5f0> (a > jline.internal.NonBlockingInputStream) > at > jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) > - locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 >nid=0x1dbe9 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000 >nid=0x1dbe8 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000 >nid=0x1dbe7 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800 >nid=0x1dbe6 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800 >nid=0x1dbe5 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000 >nid=0x1dbe4 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000 >nid=0x1dbe3 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000 >nid=0x1dbe2 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000 >nid=0x1dbe1 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800 >nid=0x1dbe0 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800 >nid=0x1dbdf waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800 >nid=0x1dbde waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000 >nid=0x1dbdd waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000 >nid=0x1dbdc runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in >Object.wait() [0x00007f4c036f5000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > - locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > > Locked ownable synchronizers: > - None > >"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000 >nid=0x1dbda in Object.wait() [0x00007f4c037f6000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > - locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > > Locked ownable synchronizers: > - None > >"main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb in Object.wait() >[0x00007f4c551a8000] > java.lang.Thread.State: BLOCKED (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396) > - locked <0x0000000792eb2c68> (a java.lang.UNIXProcess) > at > jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:199) > at > jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173) > at > jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168) > at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76) > at jline.UnixTerminal.enableInterruptCharacter(UnixTerminal.java:122) > at jline.console.ConsoleReader.readLine(ConsoleReader.java:2859) > at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126) > at sqlline.SqlLine.begin(SqlLine.java:621) > at sqlline.SqlLine.start(SqlLine.java:375) > at sqlline.SqlLine.main(SqlLine.java:268) > > Locked ownable synchronizers: > - None > >"VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable > >"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 >nid=0x1dbcc runnable > >"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 >nid=0x1dbcd runnable > >"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 >nid=0x1dbce runnable > >"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 >nid=0x1dbcf runnable > >"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 >nid=0x1dbd0 runnable > >"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 >nid=0x1dbd1 runnable > >"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 >nid=0x1dbd2 runnable > >"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 >nid=0x1dbd3 runnable > >"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 >nid=0x1dbd4 runnable > >"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 >nid=0x1dbd5 runnable > >"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 >nid=0x1dbd6 runnable > >"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 >nid=0x1dbd7 runnable > >"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 >nid=0x1dbd8 runnable > >"VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea >waiting on condition > >JNI global references: 314 > >Thread dump 3: >2016-08-11 11:02:25 >Full thread dump OpenJDK 64-Bit Server VM (25.91-b14 mixed mode): > >"Attach Listener" #1308 daemon prio=9 os_prio=0 tid=0x00007f4bfc156800 >nid=0x1cce6 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"process reaper" #1307 daemon prio=10 os_prio=0 tid=0x00007f4c4d5cd800 >nid=0x2490 waiting on condition [0x00007f4c000b6000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c00435c8> (a > java.util.concurrent.SynchronousQueue$TransferStack) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"threadDeathWatcher-2-1" #29 daemon prio=1 os_prio=0 tid=0x00007f4c4d52f800 >nid=0x1dc04 waiting on condition [0x00007f4bed9d6000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Client-1" #27 daemon prio=10 os_prio=0 tid=0x00007f4c4d493000 nid=0x1dc03 >runnable [0x00007f4bedad7000] > java.lang.Thread.State: RUNNABLE > at io.netty.channel.epoll.Native.epollWait0(Native Method) > at io.netty.channel.epoll.Native.epollWait(Native.java:148) > at > io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180) > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Curator-ServiceCache-0" #26 daemon prio=5 os_prio=0 tid=0x00007f4c4d552800 >nid=0x1dc01 waiting on condition [0x00007f4c005c1000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c0013c68> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Curator-Framework-0" #25 daemon prio=5 os_prio=0 tid=0x00007f4c4d381000 >nid=0x1dc00 waiting on condition [0x00007f4c006c2000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c0023678> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:70) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - <0x00000006c00237b8> (a >java.util.concurrent.ThreadPoolExecutor$Worker) > >"main-EventThread" #24 daemon prio=5 os_prio=0 tid=0x00007f4c4d35e800 >nid=0x1dbff waiting on condition [0x00007f4c007c3000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c001bff8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at >org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) > > Locked ownable synchronizers: > - None > >"main-SendThread(10.35.80.60:2181)" #23 daemon prio=5 os_prio=0 >tid=0x00007f4c4d420000 nid=0x1dbfe runnable [0x00007f4c008c4000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked <0x00000006c002b708> (a sun.nio.ch.Util$2) > - locked <0x00000006c002b718> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000006c002b6c0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) > at >org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) > > Locked ownable synchronizers: > - None > >"Curator-ConnectionStateManager-0" #22 daemon prio=5 os_prio=0 >tid=0x00007f4c4d32c800 nid=0x1dbfd waiting on condition [0x00007f4c009c5000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000006c0033630> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) > at > org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245) > at > org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43) > at > org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - <0x00000006c0033750> (a >java.util.concurrent.ThreadPoolExecutor$Worker) > >"NonBlockingInputStreamThread" #20 daemon prio=5 os_prio=0 >tid=0x00007f4c4cdab800 nid=0x1dbfc in Object.wait() [0x00007f4c01b52000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000006c003b5f0> (a > jline.internal.NonBlockingInputStream) > at > jline.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278) > - locked <0x00000006c003b5f0> (a jline.internal.NonBlockingInputStream) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > >"Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007f4c4c10d000 >nid=0x1dbe9 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f4c4c10a000 >nid=0x1dbe8 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f4c4c108000 >nid=0x1dbe7 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f4c4c105800 >nid=0x1dbe6 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f4c4c103800 >nid=0x1dbe5 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f4c4c101000 >nid=0x1dbe4 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ff000 >nid=0x1dbe3 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fd000 >nid=0x1dbe2 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f4c4c0fb000 >nid=0x1dbe1 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f8800 >nid=0x1dbe0 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4c4c0f6800 >nid=0x1dbdf waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4c4c0ec800 >nid=0x1dbde waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4c4c0df000 >nid=0x1dbdd waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4c4c0dd000 >nid=0x1dbdc runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > >"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4c4c0b3000 nid=0x1dbdb in >Object.wait() [0x00007f4c036f5000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > - locked <0x00000006c0053668> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > > Locked ownable synchronizers: > - None > >"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4c4c0a9000 >nid=0x1dbda in Object.wait() [0x00007f4c037f6000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > - locked <0x00000006c004c0d0> (a java.lang.ref.Reference$Lock) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > > Locked ownable synchronizers: > - None > >"main" #1 prio=5 os_prio=0 tid=0x00007f4c4c010800 nid=0x1dbcb runnable >[0x00007f4c551a8000] > java.lang.Thread.State: RUNNABLE > at java.lang.UNIXProcess.forkAndExec(Native Method) > at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) > at java.lang.ProcessImpl.start(ProcessImpl.java:134) > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) > at java.lang.Runtime.exec(Runtime.java:620) > at java.lang.Runtime.exec(Runtime.java:485) > at > jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:183) > at > jline.internal.TerminalLineSettings.exec(TerminalLineSettings.java:173) > at > jline.internal.TerminalLineSettings.stty(TerminalLineSettings.java:168) > at jline.internal.TerminalLineSettings.set(TerminalLineSettings.java:76) > at jline.UnixTerminal.enableInterruptCharacter(UnixTerminal.java:122) > at jline.console.ConsoleReader.readLine(ConsoleReader.java:2859) > at jline.console.ConsoleReader.readLine(ConsoleReader.java:2126) > at sqlline.SqlLine.begin(SqlLine.java:621) > at sqlline.SqlLine.start(SqlLine.java:375) > at sqlline.SqlLine.main(SqlLine.java:268) > > Locked ownable synchronizers: > - None > >"VM Thread" os_prio=0 tid=0x00007f4c4c09f000 nid=0x1dbd9 runnable > >"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4c4c025800 >nid=0x1dbcc runnable > >"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4c4c027800 >nid=0x1dbcd runnable > >"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4c4c029000 >nid=0x1dbce runnable > >"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02b000 >nid=0x1dbcf runnable > >"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02c800 >nid=0x1dbd0 runnable > >"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4c4c02e800 >nid=0x1dbd1 runnable > >"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4c4c030800 >nid=0x1dbd2 runnable > >"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4c4c032000 >nid=0x1dbd3 runnable > >"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f4c4c034000 >nid=0x1dbd4 runnable > >"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f4c4c036000 >nid=0x1dbd5 runnable > >"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f4c4c037800 >nid=0x1dbd6 runnable > >"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f4c4c039800 >nid=0x1dbd7 runnable > >"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f4c4c03b000 >nid=0x1dbd8 runnable > >"VM Periodic Task Thread" os_prio=0 tid=0x00007f4c4c10f800 nid=0x1dbea >waiting on condition > >JNI global references: 314 > > >-----Original Message----- >From: Avi Haleva [mailto:[email protected]] >Sent: Thursday, August 11, 2016 14:10 >To: [email protected] >Subject: RE: Drill CPU Usage > >Both the log and out files looks clean > >Attached you can find the thread dump output (3 snapshots). >I will appreciate if one can point me to the reason the drillbits >consume constantly CPU > >BTW, the CPU usage is now a constant at 40% for a single core (double >from a week ago) > > >-----Original Message----- >From: Dechang Gu [mailto:[email protected]] >Sent: Tuesday, August 2, 2016 22:21 >To: [email protected] >Subject: Re: Drill CPU Usage > >On Tue, Aug 2, 2016 at 12:49 AM, Avi Haleva <[email protected]> wrote: > >> Hi, >> I'm in the process of evaluating Drill as a analytic repositiory. >> I've noticed that even when idle, the drillbit process consume 20% of >> a single core constantly. >> >> Is that expected behavior or have I miss configured something. >> > >I don't think that is expected. I checked my system, the drillbit CPU usage >stays at 0% almost constantly when idle. >I did see occasionally it has a glitch at 20-30% CPU usage, due to zookeeper >heartbeat traffic, I guess. > >Can you collect jstack on the drillbit process to find out what is running? > >Also check drillbit.out and drillbit.log (typically under /var/log/drill/ ) to >see if there is anything suspicious. > >HTH, and Thanks, >Dechang > > >> >> I'm running drill in single node with zookeeper in a non-embeded mode >> (as I want to have more than a single connection concurrently) >> >> Thanks, >> Avi >> >> -- Oscar Morante "Self-education is, I firmly believe, the only kind of education there is." -- Isaac Asimov.
