Pig job freezes occasionally

Thu, 05 Nov 2015 04:47:14 -0800 

Hey all,

I'm experiencing a weird behaviour, when I run the script it freeze 2 out
of 3 times, and when it doesn't it takes slightly longer then Yarn. I found
that if I remove the following part from the end of the script it
significantly faster and do not freezes.

```pig
a1 = FOREACH metrics GENERATE ...;
a2 = FOREACH metrics GENERATE ...;
...
a20 = FOREACH metrics GENERATE ...;
u = UNION a1, a2, ..., a20;
STORE u INTO ... Using PigStorage('\t');
```

I use Tez 0.7.0, Pig 0.15.0, Hadoop 2.6.0. The setting pig.tez.opt.union is
disabled, an internal exception is thrown otherwise. I've attached a dump
of a frozen job, what could be the problem?

Thanks :)

Full thread dump OpenJDK 64-Bit Server VM (24.85-b03 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00007ff290018800 nid=0x104e waiting on 
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"TaskHeartbeatThread" daemon prio=10 tid=0x00007ff2aa6d5800 nid=0x6af5 waiting 
on condition [0x00007ff297127000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007701529f8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"IPC Parameter Sending Thread #0" daemon prio=10 tid=0x00007ff2aa526800 
nid=0x6ae3 waiting on condition [0x00007ff297228000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000077014de28> (a 
java.util.concurrent.SynchronousQueue$TransferStack)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"IPC Client (1726401631) connection to /10.146.179.166:54386 from 
application_1444117287785_0029" daemon prio=10 tid=0x00007ff2aa523800 
nid=0x6ae2 in Object.wait() [0x00007ff297329000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:920)
        - locked <0x000000077015de98> (a 
org.apache.hadoop.ipc.Client$Connection)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:965)

"TezChild" daemon prio=10 tid=0x00007ff2aa469000 nid=0x6ad8 waiting on 
condition [0x00007ff29762b000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at java.lang.Thread.sleep(Thread.java:340)
        at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:360)
        at 
org.apache.tez.runtime.task.ContainerReporter.callInternal(ContainerReporter.java:64)
        at 
org.apache.tez.runtime.task.ContainerReporter.callInternal(ContainerReporter.java:36)
        at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"Service Thread" daemon prio=10 tid=0x00007ff2a80d9800 nid=0x6958 runnable 
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007ff2a80d7000 nid=0x6957 waiting 
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007ff2a80d4000 nid=0x6956 waiting 
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007ff2a80d2000 nid=0x6955 runnable 
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007ff2a80a6800 nid=0x694e in Object.wait() 
[0x00007ff29dec9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0000000770157510> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        - locked <0x0000000770157510> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" daemon prio=10 tid=0x00007ff2a809e800 nid=0x694d in 
Object.wait() [0x00007ff29dfca000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0000000770157300> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:503)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
        - locked <0x0000000770157300> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007ff2a8026000 nid=0x6940 waiting on condition 
[0x00007ff2b02fc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007dd897298> (a 
com.google.common.util.concurrent.ListenableFutureTask)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
        at java.util.concurrent.FutureTask.get(FutureTask.java:187)
        at org.apache.tez.runtime.task.TezChild.run(TezChild.java:207)
        at org.apache.tez.runtime.task.TezChild.main(TezChild.java:469)

"VM Thread" prio=10 tid=0x00007ff2a809a000 nid=0x694c runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007ff2a803c000 nid=0x6941 
runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007ff2a803d800 nid=0x6942 
runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007ff2a803f800 nid=0x6943 
runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007ff2a8041800 nid=0x6944 
runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007ff2a8043800 nid=0x6945 
runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007ff2a8045000 nid=0x6946 
runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007ff2a8047000 nid=0x6947 
runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007ff2a8049000 nid=0x6948 
runnable

"VM Periodic Task Thread" prio=10 tid=0x00007ff2a80dc000 nid=0x6959 waiting on 
condition

JNI global references: 258
Full thread dump OpenJDK 64-Bit Server VM (24.85-b03 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00007f7dee91e800 nid=0x24e2 waiting on 
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"App Shared Pool - #30" daemon prio=10 tid=0x0000000001bed800 nid=0x7263 
waiting on condition [0x00007f7dcde15000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073e8d65b0> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
        at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.getSourceSpecFor(VertexImpl.java:4721)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.getOutputSpecList(VertexImpl.java:4710)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.createRemoteTaskSpec(VertexImpl.java:1472)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.scheduleTasks(VertexImpl.java:1506)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerPluginContextImpl.scheduleVertexTasks(VertexManager.java:206)
        - locked <0x000000074147ba60> (a 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerPluginContextImpl)
        at 
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:677)
        at 
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:771)
        at 
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.onSourceTaskCompleted(ShuffleVertexManager.java:500)
        - locked <0x000000074147bc58> (a 
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigGraceShuffleVertexManager)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventSourceTaskCompleted.invoke(VertexManager.java:601)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:656)
        - locked <0x000000074147bbb0> (a 
org.apache.tez.dag.app.dag.impl.VertexManager)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:651)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:651)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:640)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"App Shared Pool - #27" daemon prio=10 tid=0x0000000002556000 nid=0x5f6f 
waiting on condition [0x00007f7dcea21000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073eb6fc40> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
        at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.getTotalTasks(VertexImpl.java:1028)
        at 
org.apache.tez.dag.app.dag.impl.Edge$EdgeManagerPluginContextImpl.getSourceVertexNumTasks(Edge.java:91)
        at 
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager$CustomShuffleEdgeManager.initialize(ShuffleVertexManager.java:197)
        at org.apache.tez.dag.app.dag.impl.Edge.initialize(Edge.java:162)
        at org.apache.tez.dag.app.dag.impl.Edge.setEdgeProperty(Edge.java:178)
        - locked <0x000000073ffc3738> (a org.apache.tez.dag.app.dag.impl.Edge)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.setParallelism(VertexImpl.java:1728)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.reconfigureVertex(VertexImpl.java:1527)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerPluginContextImpl.reconfigureVertex(VertexManager.java:185)
        - locked <0x000000073ffe6070> (a 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerPluginContextImpl)
        at 
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.determineParallelismAndApply(ShuffleVertexManager.java:648)
        at 
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:664)
        at 
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:771)
        at 
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.handleVertexStateUpdate(ShuffleVertexManager.java:839)
        at 
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.onVertexStateUpdated(ShuffleVertexManager.java:852)
        - locked <0x000000073ffe5ff8> (a 
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigGraceShuffleVertexManager)
        at 
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigGraceShuffleVertexManager.onVertexStateUpdated(PigGraceShuffleVertexManager.java:103)
        - locked <0x000000073ffe5ff8> (a 
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigGraceShuffleVertexManager)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventOnVertexStateUpdate.invoke(VertexManager.java:571)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:656)
        - locked <0x000000073ffe6240> (a 
org.apache.tez.dag.app.dag.impl.VertexManager)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:651)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:651)
        at 
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:640)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #9" daemon prio=10 tid=0x00007f7ddc7f4800 nid=0x3b36 waiting 
on condition [0x00007f7dcdf17000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #8" daemon prio=10 tid=0x00007f7ddc7f3800 nid=0x3b34 waiting 
on condition [0x00007f7dce018000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #7" daemon prio=10 tid=0x00007f7ddc7f3000 nid=0x3b30 waiting 
on condition [0x00007f7dce119000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #6" daemon prio=10 tid=0x00007f7ddc8b9000 nid=0x3b2e waiting 
on condition [0x00007f7de01e9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #5" daemon prio=10 tid=0x00007f7ddc8b8800 nid=0x3b2b waiting 
on condition [0x00007f7dce21a000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #4" daemon prio=10 tid=0x00007f7ddc8b7000 nid=0x3b25 waiting 
on condition [0x00007f7dce31b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #3" daemon prio=10 tid=0x00007f7ddcfcd000 nid=0x3b23 waiting 
on condition [0x00007f7dce41c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #2" daemon prio=10 tid=0x00007f7ddc7cb000 nid=0x3b21 waiting 
on condition [0x00007f7dce51d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #1" daemon prio=10 tid=0x00007f7ddc15b800 nid=0x3b1e waiting 
on condition [0x00007f7dce61e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"ContainerLauncher #0" daemon prio=10 tid=0x00007f7ddc23b800 nid=0x3b1c waiting 
on condition [0x00007f7dce71f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d8417d8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"Thread-102" daemon prio=10 tid=0x00007f7de4570000 nid=0x3b17 waiting on 
condition [0x00007f7dce820000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000741738ce8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"java-sdk-http-connection-reaper" daemon prio=10 tid=0x00007f7de44cd800 
nid=0x3b16 waiting on condition [0x00007f7dced25000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at 
com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:112)

"ResponseProcessor for block 
BP-1133212022-10.69.28.41-1444117271455:blk_1073746413_7481" daemon prio=10 
tid=0x000000000189f000 nid=0x3b08 runnable [0x00007f7dcee26000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000073e200948> (a sun.nio.ch.Util$2)
        - locked <0x000000073e200938> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000073e200710> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at 
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
        at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
        at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
        at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
        at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at 
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2201)
        at 
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
        at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867)

"DataStreamer for file 
/tmp/temp-2118050631/.tez/application_1444117287785_0029/recovery/1/dag_1444117287785_0029_1.recovery
 block BP-1133212022-10.69.28.41-1444117271455:blk_1073746413_7481" daemon 
prio=10 tid=0x000000000245e800 nid=0x3b05 in Object.wait() [0x00007f7dcef27000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073e206eb8> (a java.util.LinkedList)
        at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564)
        - locked <0x000000073e206eb8> (a java.util.LinkedList)

"ResponseProcessor for block 
BP-1133212022-10.69.28.41-1444117271455:blk_1073746412_7480" daemon prio=10 
tid=0x0000000001e6e800 nid=0x3b04 runnable [0x00007f7dcf028000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000073bd29af0> (a sun.nio.ch.Util$2)
        - locked <0x000000073bd29ae0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000073bd298b8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at 
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
        at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
        at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
        at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
        at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at 
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2201)
        at 
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
        at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867)

"LeaseRenewer:[email protected]:8020" daemon prio=10 
tid=0x000000000189d800 nid=0x3b01 waiting on condition [0x00007f7dcf129000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
        at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
        at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
        at java.lang.Thread.run(Thread.java:745)

"DataStreamer for file 
/tmp/temp-2118050631/.tez/application_1444117287785_0029/recovery/1/summary 
block BP-1133212022-10.69.28.41-1444117271455:blk_1073746412_7480" daemon 
prio=10 tid=0x000000000189a800 nid=0x3b00 in Object.wait() [0x00007f7dcf22a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073e245f70> (a java.util.LinkedList)
        at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:564)
        - locked <0x000000073e245f70> (a java.util.LinkedList)

"State Change Notifier DAG: dag_1444117287785_0029_1" daemon prio=10 
tid=0x00000000019f2800 nid=0x3aff waiting on condition [0x00007f7dcf32b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073f40cd78> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
org.apache.tez.dag.app.dag.StateChangeNotifier$1.run(StateChangeNotifier.java:100)

"DestroyJavaVM" prio=10 tid=0x00007f7dec025000 nid=0x3a6d waiting on condition 
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Timer-1" daemon prio=10 tid=0x00007f7dee87a800 nid=0x3af9 in Object.wait() 
[0x00007f7dd437b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073e191940> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x000000073e191940> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"TaskSchedulerEventHandlerThread" prio=10 tid=0x00007f7ddc7fc000 nid=0x3af8 
waiting on condition [0x00007f7dcf42c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073bd5e3b8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
org.apache.tez.dag.app.rm.TaskSchedulerEventHandler$1.run(TaskSchedulerEventHandler.java:370)

"DelayedContainerManager" prio=10 tid=0x00007f7ddc7fa000 nid=0x3af7 in 
Object.wait() [0x00007f7dcf52d000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073f40d0f8> (a 
org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager)
        at java.lang.Object.wait(Object.java:503)
        at 
org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager.run(YarnTaskSchedulerService.java:1839)
        - locked <0x000000073f40d0f8> (a 
org.apache.tez.dag.app.rm.YarnTaskSchedulerService$DelayedContainerManager)

"TaskSchedulerAppCaller #0" daemon prio=10 tid=0x00007f7ddc7f9000 nid=0x3af6 
waiting on condition [0x00007f7dcf62e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073eb6fc40> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
        at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.getProgress(VertexImpl.java:1145)
        at org.apache.tez.dag.app.dag.impl.DAGImpl.getProgress(DAGImpl.java:778)
        at 
org.apache.tez.dag.app.DAGAppMaster.getProgress(DAGAppMaster.java:1139)
        at 
org.apache.tez.dag.app.rm.TaskSchedulerEventHandler.getProgress(TaskSchedulerEventHandler.java:556)
        at 
org.apache.tez.dag.app.rm.TaskSchedulerAppCallbackWrapper$GetProgressCallable.call(TaskSchedulerAppCallbackWrapper.java:291)
        at 
org.apache.tez.dag.app.rm.TaskSchedulerAppCallbackWrapper$GetProgressCallable.call(TaskSchedulerAppCallbackWrapper.java:282)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"AMRM Heartbeater thread" prio=10 tid=0x00007f7ddc798800 nid=0x3af5 waiting on 
condition [0x00007f7dcf72f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:248)

"IPC Client (812255777) connection to 
ip-10-69-28-41.ec2.internal/10.69.28.41:8030 from hadoop" daemon prio=10 
tid=0x00007f7ddc793800 nid=0x3af3 in Object.wait() [0x00007f7dcf830000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073db4e840> (a 
org.apache.hadoop.ipc.Client$Connection)
        at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:920)
        - locked <0x000000073db4e840> (a 
org.apache.hadoop.ipc.Client$Connection)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:965)

"AMRM Callback Handler Thread" daemon prio=10 tid=0x00007f7ddc0b1000 nid=0x3af2 
waiting on condition [0x00007f7dd447c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074a165ce8> (a 
java.util.concurrent.FutureTask)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
        at java.util.concurrent.FutureTask.get(FutureTask.java:187)
        at 
org.apache.tez.dag.app.rm.TaskSchedulerAppCallbackWrapper.getProgress(TaskSchedulerAppCallbackWrapper.java:104)
        at 
org.apache.tez.dag.app.rm.YarnTaskSchedulerService.getProgress(YarnTaskSchedulerService.java:888)
        at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:295)

"com.google.inject.internal.util.$Finalizer" daemon prio=10 
tid=0x0000000001f0f000 nid=0x3af1 in Object.wait() [0x00007f7dcf931000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073d9e6eb8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        - locked <0x000000073d9e6eb8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)

"Timer-0" daemon prio=10 tid=0x0000000001e90800 nid=0x3af0 in Object.wait() 
[0x00007f7dcfa32000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073d946620> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x000000073d946620> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"297305432@qtp-1392038805-1 - Acceptor0 
[email protected]:39873" daemon prio=10 
tid=0x0000000001ec3000 nid=0x3aec runnable [0x00007f7dcfc34000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000073d938568> (a sun.nio.ch.Util$2)
        - locked <0x000000073d938558> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000073d938180> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at 
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
        at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
        at 
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at 
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
        at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1039213249@qtp-1392038805-0" daemon prio=10 tid=0x0000000001ec6800 nid=0x3aeb 
in Object.wait() [0x00007f7dd477f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073d938878> (a 
org.mortbay.thread.QueuedThreadPool$PoolThread)
        at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
        - locked <0x000000073d938878> (a 
org.mortbay.thread.QueuedThreadPool$PoolThread)

"IPC Server handler 59 on 54386" daemon prio=10 tid=0x0000000001bd7000 
nid=0x3aea waiting on condition [0x00007f7dcfd35000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 58 on 54386" daemon prio=10 tid=0x0000000001bd2000 
nid=0x3ae9 waiting on condition [0x00007f7dcfe36000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 57 on 54386" daemon prio=10 tid=0x0000000001bcd800 
nid=0x3ae8 waiting on condition [0x00007f7dcff37000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 56 on 54386" daemon prio=10 tid=0x0000000001bb2800 
nid=0x3ae7 waiting on condition [0x00007f7dd0038000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 55 on 54386" daemon prio=10 tid=0x0000000001bab800 
nid=0x3ae6 waiting on condition [0x00007f7dd0139000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 54 on 54386" daemon prio=10 tid=0x0000000001bbd800 
nid=0x3ae5 waiting on condition [0x00007f7dd023a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 53 on 54386" daemon prio=10 tid=0x0000000001ddc800 
nid=0x3ae4 waiting on condition [0x00007f7dd033b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 52 on 54386" daemon prio=10 tid=0x0000000001bb4000 
nid=0x3ae3 waiting on condition [0x00007f7dd043c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 51 on 54386" daemon prio=10 tid=0x0000000001df2800 
nid=0x3ae2 waiting on condition [0x00007f7dd053d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 50 on 54386" daemon prio=10 tid=0x0000000001bc4000 
nid=0x3ae1 waiting on condition [0x00007f7dd063e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 49 on 54386" daemon prio=10 tid=0x0000000001bbb000 
nid=0x3ae0 waiting on condition [0x00007f7dd073f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 48 on 54386" daemon prio=10 tid=0x0000000001bb5800 
nid=0x3adf waiting on condition [0x00007f7dd0840000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 47 on 54386" daemon prio=10 tid=0x0000000001bae800 
nid=0x3ade waiting on condition [0x00007f7dd0941000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 46 on 54386" daemon prio=10 tid=0x0000000001dde000 
nid=0x3add waiting on condition [0x00007f7dd0a42000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 45 on 54386" daemon prio=10 tid=0x0000000001dda000 
nid=0x3adc waiting on condition [0x00007f7dd0b43000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 44 on 54386" daemon prio=10 tid=0x0000000001dd8000 
nid=0x3adb waiting on condition [0x00007f7dd0c44000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 43 on 54386" daemon prio=10 tid=0x0000000001df4800 
nid=0x3ada waiting on condition [0x00007f7dd0d45000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 42 on 54386" daemon prio=10 tid=0x0000000001de6800 
nid=0x3ad9 waiting on condition [0x00007f7dd0e46000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 41 on 54386" daemon prio=10 tid=0x0000000001dee800 
nid=0x3ad8 waiting on condition [0x00007f7dd0f47000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 40 on 54386" daemon prio=10 tid=0x0000000001ded800 
nid=0x3ad7 waiting on condition [0x00007f7dd1048000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 39 on 54386" daemon prio=10 tid=0x0000000001de5000 
nid=0x3ad6 waiting on condition [0x00007f7dd1149000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 38 on 54386" daemon prio=10 tid=0x0000000001dd6000 
nid=0x3ad5 waiting on condition [0x00007f7dd124a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 37 on 54386" daemon prio=10 tid=0x0000000001dfb800 
nid=0x3ad4 waiting on condition [0x00007f7dd134b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 36 on 54386" daemon prio=10 tid=0x0000000001dfc800 
nid=0x3ad3 waiting on condition [0x00007f7dd144c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 35 on 54386" daemon prio=10 tid=0x0000000001df9000 
nid=0x3ad2 waiting on condition [0x00007f7dd154d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 34 on 54386" daemon prio=10 tid=0x0000000001de1800 
nid=0x3ad1 waiting on condition [0x00007f7dd164e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 33 on 54386" daemon prio=10 tid=0x0000000001e11000 
nid=0x3ad0 waiting on condition [0x00007f7dd174f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 32 on 54386" daemon prio=10 tid=0x0000000001e0e800 
nid=0x3acf waiting on condition [0x00007f7dd1850000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 31 on 54386" daemon prio=10 tid=0x0000000001e00000 
nid=0x3ace waiting on condition [0x00007f7dd1951000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 30 on 54386" daemon prio=10 tid=0x0000000001dfd800 
nid=0x3acd waiting on condition [0x00007f7dd1a52000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 29 on 54386" daemon prio=10 tid=0x0000000001df7000 
nid=0x3acc waiting on condition [0x00007f7dd1b53000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 28 on 54386" daemon prio=10 tid=0x0000000001df0000 
nid=0x3acb waiting on condition [0x00007f7dd1c54000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 27 on 54386" daemon prio=10 tid=0x0000000001dea800 
nid=0x3aca waiting on condition [0x00007f7dd4a82000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 26 on 54386" daemon prio=10 tid=0x0000000001de2800 
nid=0x3ac9 waiting on condition [0x00007f7dd1d55000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 25 on 54386" daemon prio=10 tid=0x0000000001a5b800 
nid=0x3ac8 waiting on condition [0x00007f7dd1e56000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 24 on 54386" daemon prio=10 tid=0x0000000001dd2800 
nid=0x3ac7 waiting on condition [0x00007f7dd1f57000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 23 on 54386" daemon prio=10 tid=0x0000000001dc8000 
nid=0x3ac6 waiting on condition [0x00007f7dd2058000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 22 on 54386" daemon prio=10 tid=0x0000000001dc6000 
nid=0x3ac5 waiting on condition [0x00007f7dd2159000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 21 on 54386" daemon prio=10 tid=0x0000000001a61800 
nid=0x3ac4 waiting on condition [0x00007f7dd225a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 20 on 54386" daemon prio=10 tid=0x0000000001a53000 
nid=0x3ac3 waiting on condition [0x00007f7dd235b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 19 on 54386" daemon prio=10 tid=0x0000000001dc9800 
nid=0x3ac2 waiting on condition [0x00007f7dd245c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 18 on 54386" daemon prio=10 tid=0x0000000001d77800 
nid=0x3ac1 waiting on condition [0x00007f7dd255d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 17 on 54386" daemon prio=10 tid=0x0000000001dcf800 
nid=0x3ac0 waiting on condition [0x00007f7dd265e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 16 on 54386" daemon prio=10 tid=0x0000000001dd0800 
nid=0x3abf waiting on condition [0x00007f7dd275f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 15 on 54386" daemon prio=10 tid=0x0000000001dcd800 
nid=0x3abe waiting on condition [0x00007f7dd2860000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 14 on 54386" daemon prio=10 tid=0x0000000001dcb000 
nid=0x3abd waiting on condition [0x00007f7dd2961000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 13 on 54386" daemon prio=10 tid=0x0000000001a62800 
nid=0x3abc waiting on condition [0x00007f7dd2a62000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 12 on 54386" daemon prio=10 tid=0x0000000001a5e800 
nid=0x3abb waiting on condition [0x00007f7dd2b63000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 11 on 54386" daemon prio=10 tid=0x0000000001a58000 
nid=0x3aba waiting on condition [0x00007f7dd2c64000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 10 on 54386" daemon prio=10 tid=0x0000000001a55800 
nid=0x3ab9 waiting on condition [0x00007f7dd2d65000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 9 on 54386" daemon prio=10 tid=0x0000000001a50000 
nid=0x3ab8 waiting on condition [0x00007f7dd2e66000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 8 on 54386" daemon prio=10 tid=0x0000000001a51000 
nid=0x3ab7 waiting on condition [0x00007f7dd2f67000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 7 on 54386" daemon prio=10 tid=0x0000000001d6b000 
nid=0x3ab6 waiting on condition [0x00007f7dd3068000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 6 on 54386" daemon prio=10 tid=0x0000000001a4e000 
nid=0x3ab5 waiting on condition [0x00007f7dd3169000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 5 on 54386" daemon prio=10 tid=0x0000000001a4b800 
nid=0x3ab4 waiting on condition [0x00007f7dd326a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 4 on 54386" daemon prio=10 tid=0x0000000001d75000 
nid=0x3ab3 waiting on condition [0x00007f7dd336b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 3 on 54386" daemon prio=10 tid=0x0000000001d6d800 
nid=0x3ab2 waiting on condition [0x00007f7dd346c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 2 on 54386" daemon prio=10 tid=0x0000000001ae3800 
nid=0x3ab1 waiting on condition [0x00007f7dd356d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 1 on 54386" daemon prio=10 tid=0x0000000001ad9000 
nid=0x3ab0 waiting on condition [0x00007f7dd366e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server handler 0 on 53465" daemon prio=10 tid=0x00007f7ddc4f3800 
nid=0x3aaf waiting on condition [0x00007f7dd376e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073eb6fc40> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
        at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.getVertexProgress(VertexImpl.java:1156)
        at 
org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:817)
        at 
org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:855)
        at 
org.apache.tez.dag.api.client.DAGClientHandler.getDAGStatus(DAGClientHandler.java:67)
        at 
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.getDAGStatus(DAGClientAMProtocolBlockingPBServerImpl.java:99)
        at 
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7465)
        at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

"IPC Server handler 0 on 54386" daemon prio=10 tid=0x0000000001d76000 
nid=0x3aae waiting on condition [0x00007f7dd3870000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d7721e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

"IPC Server listener on 54386" daemon prio=10 tid=0x0000000001d70000 nid=0x3aad 
runnable [0x00007f7dd3971000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000073d7ca678> (a sun.nio.ch.Util$2)
        - locked <0x000000073d7ca668> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000073d7ca2a0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at org.apache.hadoop.ipc.Server$Listener.run(Server.java:680)

"IPC Server Responder" daemon prio=10 tid=0x0000000001d6e800 nid=0x3aac 
runnable [0x00007f7dd3a72000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000073d7c9b28> (a sun.nio.ch.Util$2)
        - locked <0x000000073d7c9b18> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000073d7936b0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:847)
        at org.apache.hadoop.ipc.Server$Responder.run(Server.java:830)

"IPC Server listener on 53465" daemon prio=10 tid=0x00007f7ddc4ff800 nid=0x3aab 
runnable [0x00007f7dd3b73000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000073d8f5280> (a sun.nio.ch.Util$2)
        - locked <0x000000073d8f5270> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000073d8f4ea8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at org.apache.hadoop.ipc.Server$Listener.run(Server.java:680)

"IPC Server Responder" daemon prio=10 tid=0x00007f7ddc4b1000 nid=0x3aaa 
runnable [0x00007f7dd3c74000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000073d90fab8> (a sun.nio.ch.Util$2)
        - locked <0x000000073d90faa8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000073d90f990> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:847)
        at org.apache.hadoop.ipc.Server$Responder.run(Server.java:830)

"IPC Server idle connection scanner for port 53465" daemon prio=10 
tid=0x00007f7ddc04a800 nid=0x3aa9 in Object.wait() [0x00007f7dd3d75000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073d8f4990> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x000000073d8f4990> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"IPC Server idle connection scanner for port 54386" daemon prio=10 
tid=0x0000000001ba0800 nid=0x3aa8 in Object.wait() [0x00007f7dd3e76000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073d7c9d88> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x000000073d7c9d88> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Socket Reader #1 for port 53465" prio=10 tid=0x00007f7ddc222000 nid=0x3aa7 
runnable [0x00007f7dd4b83000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000073d8f46a0> (a sun.nio.ch.Util$2)
        - locked <0x000000073d8f4690> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000073d8f4568> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at 
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:628)
        - locked <0x000000073d8f5588> (a 
org.apache.hadoop.ipc.Server$Listener$Reader)
        at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:607)

"Socket Reader #1 for port 54386" prio=10 tid=0x0000000001a64800 nid=0x3aa6 
runnable [0x00007f7dd4179000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x000000073d7c9928> (a sun.nio.ch.Util$2)
        - locked <0x000000073d7c9918> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000073d7937d8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at 
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:628)
        - locked <0x000000073d7ca980> (a 
org.apache.hadoop.ipc.Server$Listener$Reader)
        at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:607)

"ContainerLauncher Event Handler" prio=10 tid=0x0000000001980800 nid=0x3aa5 
waiting on condition [0x00007f7dd4078000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073bdfc308> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
org.apache.tez.dag.app.launcher.ContainerLauncherImpl$1.run(ContainerLauncherImpl.java:292)

"RecoveryEventHandlingThread" prio=10 tid=0x00007f7ddc1d6000 nid=0x3aa4 waiting 
on condition [0x00007f7dd4981000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d9734b8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
org.apache.tez.dag.history.recovery.RecoveryService$1.run(RecoveryService.java:164)
        at java.lang.Thread.run(Thread.java:745)

"HistoryEventHandlingThread" prio=10 tid=0x00007f7ddc582000 nid=0x3aa3 waiting 
on condition [0x00007f7dd4880000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073d973908> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.getEventBatch(ATSHistoryLoggingService.java:248)
        at 
org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService.access$600(ATSHistoryLoggingService.java:52)
        at 
org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService$1.run(ATSHistoryLoggingService.java:174)
        - locked <0x000000073d974598> (a java.lang.Object)
        at java.lang.Thread.run(Thread.java:745)

"ContainerHeartbeatHandler PingChecker" prio=10 tid=0x00007f7dee86d800 
nid=0x3aa2 waiting on condition [0x00007f7dd457d000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.tez.dag.app.HeartbeatHandlerBase$PingChecker.run(HeartbeatHandlerBase.java:159)
        at java.lang.Thread.run(Thread.java:745)

"TaskHeartbeatHandler PingChecker" prio=10 tid=0x00007f7de4007000 nid=0x3aa1 
waiting on condition [0x00007f7dd467e000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.tez.dag.app.HeartbeatHandlerBase$PingChecker.run(HeartbeatHandlerBase.java:159)
        at java.lang.Thread.run(Thread.java:745)

"Dispatcher thread: Speculator" prio=10 tid=0x0000000001b60800 nid=0x3aa0 
waiting on condition [0x00007f7dd3f77000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073bd2a3a0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
        at java.lang.Thread.run(Thread.java:745)

"Dispatcher thread: Central" prio=10 tid=0x0000000001d7c800 nid=0x3a9d waiting 
on condition [0x00007f7dd427a000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073eb6fc40> (a 
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1809)
        at 
org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:198)
        at 
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1943)
        at 
org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1929)
        at 
org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
        at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
        at java.lang.Thread.run(Thread.java:745)

"IPC Parameter Sending Thread #0" daemon prio=10 tid=0x00007f7dee670800 
nid=0x3a8e waiting on condition [0x00007f7dd4c84000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000073bc795f0> (a 
java.util.concurrent.SynchronousQueue$TransferStack)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

"Thread-5" daemon prio=10 tid=0x00007f7dee5f1000 nid=0x3a8c runnable 
[0x00007f7de02f9000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
        at 
org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
        at 
org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:459)
        at java.lang.Thread.run(Thread.java:745)

"Service Thread" daemon prio=10 tid=0x00007f7dec0d1000 nid=0x3a7c runnable 
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f7dec0ce800 nid=0x3a7b waiting 
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f7dec0cb800 nid=0x3a7a waiting 
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f7dec0ca000 nid=0x3a79 runnable 
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f7dec09f800 nid=0x3a78 in Object.wait() 
[0x00007f7de2661000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073bf3eec8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        - locked <0x000000073bf3eec8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" daemon prio=10 tid=0x00007f7dec09e000 nid=0x3a77 in 
Object.wait() [0x00007f7de2762000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000073bd38500> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:503)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
        - locked <0x000000073bd38500> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00007f7dec099800 nid=0x3a76 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f7dec03b000 nid=0x3a6e 
runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f7dec03d000 nid=0x3a6f 
runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f7dec03f000 nid=0x3a70 
runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f7dec040800 nid=0x3a71 
runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f7dec042800 nid=0x3a72 
runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f7dec044800 nid=0x3a73 
runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f7dec046000 nid=0x3a74 
runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f7dec048000 nid=0x3a75 
runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f7dec0d3800 nid=0x3a7d waiting on 
condition

JNI global references: 242

Reply via email to