[jira] [Commented] (CRAIL-98) Crail, used as temporary results storage (type2 storage), misses close from TeraSort

David Crespi (JIRA) Wed, 26 Jun 2019 08:43:30 -0700


    [ 
https://issues.apache.org/jira/browse/CRAIL-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873441#comment-16873441
 ]


David Crespi commented on CRAIL-98:
-----------------------------------

Jonas,

I believe I built he repo correctly, but I still get the hang.  I am running 
with disni-2.1 and jnvmf-1.6 if that makes

any difference.



Crail version: v1.0-101-gf5d2dde



jstack 2765

2019-06-26 08:37:42

Full thread dump OpenJDK 64-Bit Server VM (25.212-b03 mixed mode):



"Attach Listener" #136 daemon prio=9 os_prio=0 tid=0x00007ff404001000 nid=0xc33 
waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"DestroyJavaVM" #135 prio=5 os_prio=0 tid=0x00007ff4cc013000 nid=0xb0c waiting 
on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"task-result-getter-3" #134 daemon prio=5 os_prio=0 tid=0x00007ff318004800 
nid=0xbb8 waiting on condition [0x00007ff2288f6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c1603dc0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"task-result-getter-2" #133 daemon prio=5 os_prio=0 tid=0x00007ff324001000 
nid=0xbb7 waiting on condition [0x00007ff2289f7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c1603dc0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"shuffle-server-5-4" #132 daemon prio=5 os_prio=0 tid=0x00007ff258004800 
nid=0xbb6 runnable [0x00007ff228af8000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c15bb6c8> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c15bc7c8> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c15bc6f0> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"shuffle-server-5-3" #131 daemon prio=5 os_prio=0 tid=0x00007ff258003000 
nid=0xbb5 runnable [0x00007ff228bf9000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c15bf028> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

       - locked <0x00000000c15c0128> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c15c0050> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"task-result-getter-1" #130 daemon prio=5 os_prio=0 tid=0x00007ff390006800 
nid=0xbb4 waiting on condition [0x00007ff228cfa000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c1603dc0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"task-result-getter-0" #129 daemon prio=5 os_prio=0 tid=0x00007ff390004000 
nid=0xbb3 waiting on condition [0x00007ff228dfb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c1603dc0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"shuffle-server-5-2" #128 daemon prio=5 os_prio=0 tid=0x00007ff258001800 
nid=0xbb2 runnable [0x00007ff24413c000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c15ea990> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c15eba90> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c15eb9b8> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-8" #127 daemon prio=5 os_prio=0 tid=0x00007ff2d000f800 nid=0xbb1 
runnable [0x00007ff24423d000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c168fb90> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c1690c90> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1690bb8> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-7" #126 daemon prio=5 os_prio=0 tid=0x00007ff2d000e000 nid=0xbb0 
runnable [0x00007ff24433e000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c16934f0> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c16945f0> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1694518> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-6" #125 daemon prio=5 os_prio=0 tid=0x00007ff2d000c000 nid=0xbaf 
runnable [0x00007ff24443f000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c19550d8> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c19561d8> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1956100> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-5" #124 daemon prio=5 os_prio=0 tid=0x00007ff2d000a800 nid=0xbae 
runnable [0x00007ff244540000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c1958a38> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c1959b38> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1959a60> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-4" #123 daemon prio=5 os_prio=0 tid=0x00007ff2d0008800 nid=0xbad 
runnable [0x00007ff244641000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c19807b8> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c19818b8> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c19817e0> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"threadDeathWatcher-6-1" #122 daemon prio=1 os_prio=0 tid=0x00007ff22c003000 
nid=0xbac waiting on condition [0x00007ff244742000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

        at java.lang.Thread.sleep(Native Method)

        at 
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-3" #121 daemon prio=5 os_prio=0 tid=0x00007ff2d0007800 nid=0xbab 
runnable [0x00007ff244843000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c1984118> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c1985218> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c1985140> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-2" #120 daemon prio=5 os_prio=0 tid=0x00007ff2d0005800 nid=0xbaa 
runnable [0x00007ff244944000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c19a7a98> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c19a8b98> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c19a8ac0> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"Thread-19" #119 daemon prio=5 os_prio=0 tid=0x00007ff2406f4000 nid=0xba9 
runnable [0x00007ff245052000]

   java.lang.Thread.State: RUNNABLE

        at com.ibm.disni.verbs.impl.NativeDispatcher._getCmEvent(Native Method)

        at com.ibm.disni.verbs.impl.RdmaCmNat.getCmEvent(RdmaCmNat.java:193)

        at 
com.ibm.disni.verbs.RdmaEventChannel.getCmEvent(RdmaEventChannel.java:75)

        at com.ibm.disni.RdmaCmProcessor.run(RdmaCmProcessor.java:69)

        at java.lang.Thread.run(Thread.java:748)



"Thread-18" #118 prio=5 os_prio=0 tid=0x00007ff240575000 nid=0xba8 waiting on 
condition [0x00007ff247cfd000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

        at java.lang.Thread.sleep(Native Method)

        at 
org.apache.crail.storage.nvmf.NvmfStorageClient.lambda$new$0(NvmfStorageClient.java:62)

        at 
org.apache.crail.storage.nvmf.NvmfStorageClient$$Lambda$685/933027910.run(Unknown
 Source)

        at java.lang.Thread.run(Thread.java:748)



"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" 
#117 daemon prio=5 os_prio=0 tid=0x00007ff240539000 nid=0xba7 in Object.wait() 
[0x00007ff247dfe000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000fac10178> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000000fac10178> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)

        at 
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063)

        at java.lang.Thread.run(Thread.java:748)



"element-tracking-store-worker" #116 daemon prio=5 os_prio=0 
tid=0x00007ff254011000 nid=0xba6 waiting on condition [0x00007ff2d43db000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c02a2888> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"spark-listener-group-executorManagement" #102 daemon prio=5 os_prio=0 
tid=0x00007ff4cd77d000 nid=0xba5 waiting on condition [0x00007ff2d44dc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c052af18> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:97)

        at 
org.apache.spark.scheduler.AsyncEventQueue$$Lambda$524/101862538.apply$mcJ$sp(Unknown
 Source)

        at 
scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)

        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)

        at 
org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)

        at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:83)

        at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2$$Lambda$523/36554775.apply$mcV$sp(Unknown
 Source)

        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)

        at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:83)



"spark-listener-group-appStatus" #25 daemon prio=5 os_prio=0 
tid=0x00007ff4cd77a800 nid=0xba4 waiting on condition [0x00007ff2d45dd000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04f23f0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:97)

        at 
org.apache.spark.scheduler.AsyncEventQueue$$Lambda$524/101862538.apply$mcJ$sp(Unknown
 Source)

        at 
scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)

        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)

        at 
org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)

        at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:83)

        at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2$$Lambda$523/36554775.apply$mcV$sp(Unknown
 Source)

        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)

        at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:83)



"context-cleaner-periodic-gc" #115 daemon prio=5 os_prio=0 
tid=0x00007ff4cd778800 nid=0xba3 waiting on condition [0x00007ff2d46de000]

   java.lang.Thread.State: TIMED_WAITING (parking)

       at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000fac28178> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"Spark Context Cleaner" #114 daemon prio=5 os_prio=0 tid=0x00007ff4cd780800 
nid=0xba2 in Object.wait() [0x00007ff2d47df000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000000fbc38190> (a java.lang.ref.ReferenceQueue$Lock)

        at 
org.apache.spark.ContextCleaner.$anonfun$keepCleaning$1(ContextCleaner.scala:181)

        at 
org.apache.spark.ContextCleaner$$Lambda$519/1312241467.apply$mcV$sp(Unknown 
Source)

        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)

        at 
org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:179)

        at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)



"shuffle-server-5-1" #113 daemon prio=5 os_prio=0 tid=0x00007ff4cf48f000 
nid=0xba1 runnable [0x00007ff2d4ae0000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c0471d28> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c0471d40> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c0471ce0> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"rpc-client-1-1" #112 daemon prio=5 os_prio=0 tid=0x00007ff260005800 nid=0xba0 
runnable [0x00007ff2d5de2000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c04b5270> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c04b5288> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c04b5228> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"netty-rpc-connection-0" #111 daemon prio=5 os_prio=0 tid=0x00007ff26800d000 
nid=0xb9f waiting on condition [0x00007ff2d5ee3000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04e0970> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"appclient-registration-retry-thread" #110 daemon prio=5 os_prio=0 
tid=0x00007ff388003800 nid=0xb9e waiting on condition [0x00007ff2d5fe4000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c1606920> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"driver-revive-thread" #108 daemon prio=5 os_prio=0 tid=0x00007ff390001800 
nid=0xb9c waiting on condition [0x00007ff2d61e6000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c05003c0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dag-scheduler-event-loop" #107 daemon prio=5 os_prio=0 tid=0x00007ff4cf47c000 
nid=0xb9b waiting on condition [0x00007ff2d62e7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c0500698> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)

        at 
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)

        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:47)



"netty-rpc-env-timeout" #106 daemon prio=5 os_prio=0 tid=0x00007ff28c00b800 
nid=0xb9a waiting on condition [0x00007ff2d65e8000]

   java.lang.Thread.State: TIMED_WAITING (parking)

       at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c046f460> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"Timer-1" #105 daemon prio=5 os_prio=0 tid=0x00007ff4cf43d800 nid=0xb99 in 
Object.wait() [0x00007ff2d66e9000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000c0466818> (a java.util.TaskQueue)

        at java.lang.Object.wait(Object.java:502)

        at java.util.TimerThread.mainLoop(Timer.java:526)

        - locked <0x00000000c0466818> (a java.util.TaskQueue)

        at java.util.TimerThread.run(Timer.java:505)



"Timer-0" #104 daemon prio=5 os_prio=0 tid=0x00007ff4cf43b800 nid=0xb98 in 
Object.wait() [0x00007ff2d67ea000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000c0500ac0> (a java.util.TaskQueue)

        at java.lang.Object.wait(Object.java:502)

        at java.util.TimerThread.mainLoop(Timer.java:526)

        - locked <0x00000000c0500ac0> (a java.util.TaskQueue)

        at java.util.TimerThread.run(Timer.java:505)



"heartbeat-receiver-event-loop-thread" #103 daemon prio=5 os_prio=0 
tid=0x00007ff3a4005800 nid=0xb97 waiting on condition [0x00007ff2d69ec000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c0508a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-101" #101 daemon prio=5 os_prio=0 tid=0x00007ff28c001000 nid=0xb96 
waiting on condition [0x00007ff2d68eb000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c0510a30> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at 
org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-99-acceptor-3@4cad37e6-Spark@109152a7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}"
 #99 daemon prio=3 os_prio=0 tid=0x00007ff4cf398000 nid=0xb94 waiting for 
monitor entry [0x00007ff2d6aed000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)

        - waiting to lock <0x00000000c045ee08> (a java.lang.Object)

        at 
org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)

        at 
org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-98-acceptor-2@4287e01e-Spark@109152a7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}"
 #98 daemon prio=3 os_prio=0 tid=0x00007ff4cf396000 nid=0xb93 waiting for 
monitor entry [0x00007ff2d6bee000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)

        - waiting to lock <0x00000000c045ee08> (a java.lang.Object)

        at 
org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)

        at 
org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-97-acceptor-1@2e7f42f4-Spark@109152a7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}"
 #97 daemon prio=3 os_prio=0 tid=0x00007ff4cf394800 nid=0xb92 waiting for 
monitor entry [0x00007ff2d6cef000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)

        - waiting to lock <0x00000000c045ee08> (a java.lang.Object)

        at 
org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)

        at 
org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-96-acceptor-0@1e56af1d-Spark@109152a7{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}"
 #96 daemon prio=3 os_prio=0 tid=0x00007ff4cf392800 nid=0xb91 runnable 
[0x00007ff2d6df0000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)

        at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)

        - locked <0x00000000c045ee08> (a java.lang.Object)

        at 
org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)

        at 
org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-95" #95 daemon prio=5 os_prio=0 tid=0x00007ff4cf391000 nid=0xb90 
runnable [0x00007ff2d6ef1000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c049a898> (a sun.nio.ch.Util$3)

        - locked <0x00000000c049a8a8> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c049a850> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at 
org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)

        at 
org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)

        at 
org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-94" #94 daemon prio=5 os_prio=0 tid=0x00007ff4cf38f000 nid=0xb8f 
runnable [0x00007ff2d6ff2000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c046faa0> (a sun.nio.ch.Util$3)

        - locked <0x00000000c046fab0> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c046fa58> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at 
org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)

        at 
org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)

        at 
org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-93" #93 daemon prio=5 os_prio=0 tid=0x00007ff4cf38e000 nid=0xb8e 
runnable [0x00007ff2d70f3000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c0466cd0> (a sun.nio.ch.Util$3)

        - locked <0x00000000c0466ce0> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c0466c88> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at 
org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)

        at 
org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)

        at 
org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"SparkUI-92" #92 daemon prio=5 os_prio=0 tid=0x00007ff4cf38b000 nid=0xb8d 
runnable [0x00007ff2d71f4000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c04e0c80> (a sun.nio.ch.Util$3)

        - locked <0x00000000c04e0c90> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c04e0c38> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)

        at 
org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)

        at 
org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

        at 
org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)

        at 
org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

        at 
org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

        at java.lang.Thread.run(Thread.java:748)



"RemoteBlock-temp-file-clean-thread" #91 daemon prio=5 os_prio=0 
tid=0x00007ff4cf28d800 nid=0xb8c in Object.wait() [0x00007ff2d74f5000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000000c0528e28> (a java.lang.ref.ReferenceQueue$Lock)

        at 
org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager.org$apache$spark$storage$BlockManager$RemoteBlockDownloadFileManager$$keepCleaning(BlockManager.scala:1724)

        at 
org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager$$anon$2.run(BlockManager.scala:1692)



"map-output-dispatcher-7" #90 daemon prio=5 os_prio=0 tid=0x00007ff4cf238000 
nid=0xb8b waiting on condition [0x00007ff2d77f6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c048a040> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-6" #89 daemon prio=5 os_prio=0 tid=0x00007ff4cf236800 
nid=0xb8a waiting on condition [0x00007ff2d78f7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c048a040> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-5" #88 daemon prio=5 os_prio=0 tid=0x00007ff4cf234800 
nid=0xb89 waiting on condition [0x00007ff2d79f8000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c048a040> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-4" #87 daemon prio=5 os_prio=0 tid=0x00007ff4cf23a800 
nid=0xb88 waiting on condition [0x00007ff2d7af9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c048a040> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-3" #86 daemon prio=5 os_prio=0 tid=0x00007ff4cf239800 
nid=0xb87 waiting on condition [0x00007ff2d7bfa000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c048a040> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-2" #85 daemon prio=5 os_prio=0 tid=0x00007ff4cf22b800 
nid=0xb86 waiting on condition [0x00007ff2d7cfb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c048a040> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-1" #84 daemon prio=5 os_prio=0 tid=0x00007ff4cf23c000 
nid=0xb85 waiting on condition [0x00007ff2d7dfc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c048a040> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"map-output-dispatcher-0" #83 daemon prio=5 os_prio=0 tid=0x00007ff4cf22f800 
nid=0xb84 waiting on condition [0x00007ff2d7efd000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c048a040> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"rpc-server-3-1" #82 daemon prio=5 os_prio=0 tid=0x00007ff4cf1f5800 nid=0xb83 
runnable [0x00007ff2d7ffe000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000000c0513f10> (a 
io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000000c17734e0> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000000c0514f38> (a sun.nio.ch.EPollSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

        at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-55" #81 daemon prio=5 os_prio=0 tid=0x00007ff4cef98800 
nid=0xb82 waiting on condition [0x00007ff3b85c6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-54" #80 daemon prio=5 os_prio=0 tid=0x00007ff4cef96800 
nid=0xb81 waiting on condition [0x00007ff3b86c7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-53" #79 daemon prio=5 os_prio=0 tid=0x00007ff4cef95000 
nid=0xb80 waiting on condition [0x00007ff3b87c8000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-52" #78 daemon prio=5 os_prio=0 tid=0x00007ff4cef93000 
nid=0xb7f waiting on condition [0x00007ff3b88c9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-51" #77 daemon prio=5 os_prio=0 tid=0x00007ff4cef91800 
nid=0xb7e waiting on condition [0x00007ff3b89ca000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-50" #76 daemon prio=5 os_prio=0 tid=0x00007ff4cef8f800 
nid=0xb7d waiting on condition [0x00007ff3b8acb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-49" #75 daemon prio=5 os_prio=0 tid=0x00007ff4cef8e000 
nid=0xb7c waiting on condition [0x00007ff3b8bcc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-48" #74 daemon prio=5 os_prio=0 tid=0x00007ff4cef8c000 
nid=0xb7b waiting on condition [0x00007ff3b8ccd000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-47" #73 daemon prio=5 os_prio=0 tid=0x00007ff4cef8a800 
nid=0xb7a waiting on condition [0x00007ff3b8dce000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-46" #72 daemon prio=5 os_prio=0 tid=0x00007ff4cef89000 
nid=0xb79 waiting on condition [0x00007ff3b8ecf000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-45" #71 daemon prio=5 os_prio=0 tid=0x00007ff4cef87000 
nid=0xb78 waiting on condition [0x00007ff3b8fd0000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-44" #70 daemon prio=5 os_prio=0 tid=0x00007ff4cef85800 
nid=0xb77 waiting on condition [0x00007ff3b90d1000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-43" #69 daemon prio=5 os_prio=0 tid=0x00007ff4cef83800 
nid=0xb76 waiting on condition [0x00007ff3b91d2000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-42" #68 daemon prio=5 os_prio=0 tid=0x00007ff4cef82000 
nid=0xb75 waiting on condition [0x00007ff3b92d3000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-41" #67 daemon prio=5 os_prio=0 tid=0x00007ff4cef80000 
nid=0xb74 waiting on condition [0x00007ff3b93d4000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-40" #66 daemon prio=5 os_prio=0 tid=0x00007ff4cef7e800 
nid=0xb73 waiting on condition [0x00007ff3b94d5000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-39" #65 daemon prio=5 os_prio=0 tid=0x00007ff4cef7d000 
nid=0xb72 waiting on condition [0x00007ff3b95d6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-38" #64 daemon prio=5 os_prio=0 tid=0x00007ff4cef7b000 
nid=0xb71 waiting on condition [0x00007ff3b96d7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-37" #63 daemon prio=5 os_prio=0 tid=0x00007ff4cef79800 
nid=0xb70 waiting on condition [0x00007ff3b97d8000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-36" #62 daemon prio=5 os_prio=0 tid=0x00007ff4cef77800 
nid=0xb6f waiting on condition [0x00007ff3b98d9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-35" #61 daemon prio=5 os_prio=0 tid=0x00007ff4cef76000 
nid=0xb6e waiting on condition [0x00007ff3b99da000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-34" #60 daemon prio=5 os_prio=0 tid=0x00007ff4cef74800 
nid=0xb6d waiting on condition [0x00007ff3b9adb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-33" #59 daemon prio=5 os_prio=0 tid=0x00007ff4cef72800 
nid=0xb6c waiting on condition [0x00007ff3b9bdc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-32" #58 daemon prio=5 os_prio=0 tid=0x00007ff4cef71000 
nid=0xb6b waiting on condition [0x00007ff3b9cdd000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-31" #57 daemon prio=5 os_prio=0 tid=0x00007ff4cef6f000 
nid=0xb6a waiting on condition [0x00007ff3b9dde000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-30" #56 daemon prio=5 os_prio=0 tid=0x00007ff4cef6d800 
nid=0xb69 waiting on condition [0x00007ff3b9edf000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-29" #55 daemon prio=5 os_prio=0 tid=0x00007ff4cef6b800 
nid=0xb68 waiting on condition [0x00007ff3b9fe0000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-28" #54 daemon prio=5 os_prio=0 tid=0x00007ff4cef6a000 
nid=0xb67 waiting on condition [0x00007ff3ba0e1000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-27" #53 daemon prio=5 os_prio=0 tid=0x00007ff4cef68800 
nid=0xb66 waiting on condition [0x00007ff3ba1e2000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-26" #52 daemon prio=5 os_prio=0 tid=0x00007ff4cef66800 
nid=0xb65 waiting on condition [0x00007ff3ba2e3000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-25" #51 daemon prio=5 os_prio=0 tid=0x00007ff4cef65000 
nid=0xb64 waiting on condition [0x00007ff3ba3e4000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-24" #50 daemon prio=5 os_prio=0 tid=0x00007ff4cef63000 
nid=0xb63 waiting on condition [0x00007ff3ba4e5000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-23" #49 daemon prio=5 os_prio=0 tid=0x00007ff4cef61800 
nid=0xb62 waiting on condition [0x00007ff3ba5e6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-22" #48 daemon prio=5 os_prio=0 tid=0x00007ff4cef5f800 
nid=0xb61 waiting on condition [0x00007ff3ba6e7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-21" #47 daemon prio=5 os_prio=0 tid=0x00007ff4cef5e000 
nid=0xb60 waiting on condition [0x00007ff3ba7e8000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-20" #46 daemon prio=5 os_prio=0 tid=0x00007ff4cef5c800 
nid=0xb5f waiting on condition [0x00007ff3ba8e9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-19" #45 daemon prio=5 os_prio=0 tid=0x00007ff4cef5a800 
nid=0xb5e waiting on condition [0x00007ff3ba9ea000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-18" #44 daemon prio=5 os_prio=0 tid=0x00007ff4cef59000 
nid=0xb5d waiting on condition [0x00007ff3baaeb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-17" #43 daemon prio=5 os_prio=0 tid=0x00007ff4cef57000 
nid=0xb5c waiting on condition [0x00007ff3babec000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-16" #42 daemon prio=5 os_prio=0 tid=0x00007ff4cef55800 
nid=0xb5b waiting on condition [0x00007ff3baced000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-15" #41 daemon prio=5 os_prio=0 tid=0x00007ff4cef54000 
nid=0xb5a waiting on condition [0x00007ff3badee000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-14" #40 daemon prio=5 os_prio=0 tid=0x00007ff4cef52000 
nid=0xb59 waiting on condition [0x00007ff3baeef000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-13" #39 daemon prio=5 os_prio=0 tid=0x00007ff4cef50800 
nid=0xb58 waiting on condition [0x00007ff3baff0000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-12" #38 daemon prio=5 os_prio=0 tid=0x00007ff4cef4e800 
nid=0xb57 waiting on condition [0x00007ff3bb0f1000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-11" #37 daemon prio=5 os_prio=0 tid=0x00007ff4cef4d000 
nid=0xb56 waiting on condition [0x00007ff3bb1f2000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-10" #36 daemon prio=5 os_prio=0 tid=0x00007ff4cef4b000 
nid=0xb55 waiting on condition [0x00007ff3bb2f3000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-9" #35 daemon prio=5 os_prio=0 tid=0x00007ff4cef49800 
nid=0xb54 waiting on condition [0x00007ff3bb3f4000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-8" #34 daemon prio=5 os_prio=0 tid=0x00007ff4cef48000 
nid=0xb53 waiting on condition [0x00007ff3bb4f5000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-7" #33 daemon prio=5 os_prio=0 tid=0x00007ff4cef46000 
nid=0xb52 waiting on condition [0x00007ff3bb5f6000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-6" #32 daemon prio=5 os_prio=0 tid=0x00007ff4cef44800 
nid=0xb51 waiting on condition [0x00007ff3bb6f7000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-5" #31 daemon prio=5 os_prio=0 tid=0x00007ff4cef42800 
nid=0xb50 waiting on condition [0x00007ff3bb7f8000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-4" #30 daemon prio=5 os_prio=0 tid=0x00007ff4cef41000 
nid=0xb4f waiting on condition [0x00007ff3bb8f9000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-3" #29 daemon prio=5 os_prio=0 tid=0x00007ff4cef3d000 
nid=0xb4e waiting on condition [0x00007ff3bb9fa000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-2" #28 daemon prio=5 os_prio=0 tid=0x00007ff4cef3b800 
nid=0xb4d waiting on condition [0x00007ff3bbafb000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-1" #27 daemon prio=5 os_prio=0 tid=0x00007ff4cef3a000 
nid=0xb4c waiting on condition [0x00007ff3bbbfc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"dispatcher-event-loop-0" #26 daemon prio=5 os_prio=0 tid=0x00007ff4cef39000 
nid=0xb4b waiting on condition [0x00007ff3c03dc000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c04d0800> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at 
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"process reaper" #23 daemon prio=10 os_prio=0 tid=0x00007ff4ced75800 nid=0xb49 
waiting on condition [0x00007ff418068000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000000c0026488> (a 
java.util.concurrent.SynchronousQueue$TransferStack)

        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)

        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)

        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)

        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)



"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007ff4cc126800 nid=0xb46 
runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007ff4cc123800 
nid=0xb45 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007ff4cc121800 
nid=0xb44 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007ff4cc11f000 
nid=0xb43 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007ff4cc11d800 
nid=0xb42 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007ff4cc11b800 
nid=0xb41 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007ff4cc119000 
nid=0xb40 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007ff4cc117000 
nid=0xb3f waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007ff4cc115000 
nid=0xb3e waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007ff4cc113000 
nid=0xb3d waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007ff4cc110800 
nid=0xb3c waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007ff4cc106800 
nid=0xb3b waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ff4cc104800 
nid=0xb3a waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ff4cc100000 
nid=0xb39 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ff4cc0fe800 
nid=0xb38 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ff4cc0fb800 
nid=0xb37 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ff4cc0f9800 nid=0xb36 
runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE



"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ff4cc0d2800 nid=0xb35 in 
Object.wait() [0x00007ff41a1e0000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000c001ec38> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000000c001ec38> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)

        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)



"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ff4cc0d0000 
nid=0xb34 in Object.wait() [0x00007ff41a2e1000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000c009e388> (a java.lang.ref.Reference$Lock)

        at java.lang.Object.wait(Object.java:502)

        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)

        - locked <0x00000000c009e388> (a java.lang.ref.Reference$Lock)

        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)



"VM Thread" os_prio=0 tid=0x00007ff4cc0c6000 nid=0xb33 runnable



"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ff4cc029000 nid=0xb0d 
runnable



"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007ff4cc02a800 nid=0xb0e 
runnable



"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007ff4cc02c800 nid=0xb0f 
runnable



"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007ff4cc02e000 nid=0xb10 
runnable



"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007ff4cc030000 nid=0xb11 
runnable



"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007ff4cc031800 nid=0xb12 
runnable



"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007ff4cc033800 nid=0xb13 
runnable



"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007ff4cc035000 nid=0xb14 
runnable



"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007ff4cc037000 nid=0xb15 
runnable



"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007ff4cc038800 nid=0xb16 
runnable



"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007ff4cc03a800 nid=0xb17 
runnable



"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007ff4cc03c000 nid=0xb18 
runnable



"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007ff4cc03e000 nid=0xb19 
runnable



"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007ff4cc03f800 nid=0xb1a 
runnable



"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007ff4cc041800 nid=0xb1b 
runnable



"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007ff4cc043000 nid=0xb1c 
runnable



"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007ff4cc045000 nid=0xb1d 
runnable



"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007ff4cc046800 nid=0xb1e 
runnable



"GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007ff4cc048800 nid=0xb1f 
runnable



"GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007ff4cc04a000 nid=0xb20 
runnable



"GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007ff4cc04c000 nid=0xb21 
runnable



"GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007ff4cc04d800 nid=0xb22 
runnable



"GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007ff4cc04f800 nid=0xb23 
runnable



"GC task thread#23 (ParallelGC)" os_prio=0 tid=0x00007ff4cc051000 nid=0xb24 
runnable



"GC task thread#24 (ParallelGC)" os_prio=0 tid=0x00007ff4cc053000 nid=0xb25 
runnable



"GC task thread#25 (ParallelGC)" os_prio=0 tid=0x00007ff4cc054800 nid=0xb26 
runnable



"GC task thread#26 (ParallelGC)" os_prio=0 tid=0x00007ff4cc056800 nid=0xb27 
runnable



"GC task thread#27 (ParallelGC)" os_prio=0 tid=0x00007ff4cc058000 nid=0xb28 
runnable



"GC task thread#28 (ParallelGC)" os_prio=0 tid=0x00007ff4cc05a000 nid=0xb29 
runnable



"GC task thread#29 (ParallelGC)" os_prio=0 tid=0x00007ff4cc05b800 nid=0xb2a 
runnable



"GC task thread#30 (ParallelGC)" os_prio=0 tid=0x00007ff4cc05d800 nid=0xb2b 
runnable



"GC task thread#31 (ParallelGC)" os_prio=0 tid=0x00007ff4cc05f000 nid=0xb2c 
runnable



"GC task thread#32 (ParallelGC)" os_prio=0 tid=0x00007ff4cc061000 nid=0xb2d 
runnable



"GC task thread#33 (ParallelGC)" os_prio=0 tid=0x00007ff4cc062800 nid=0xb2e 
runnable



"GC task thread#34 (ParallelGC)" os_prio=0 tid=0x00007ff4cc064800 nid=0xb2f 
runnable



"GC task thread#35 (ParallelGC)" os_prio=0 tid=0x00007ff4cc066000 nid=0xb30 
runnable



"GC task thread#36 (ParallelGC)" os_prio=0 tid=0x00007ff4cc068000 nid=0xb31 
runnable



"GC task thread#37 (ParallelGC)" os_prio=0 tid=0x00007ff4cc069800 nid=0xb32 
runnable



"VM Periodic Task Thread" os_prio=0 tid=0x00007ff4cc129000 nid=0xb47 waiting on 
condition



JNI global references: 2259





Regards,



           David







> Crail, used as temporary results storage (type2 storage), misses close from 
> TeraSort
> ------------------------------------------------------------------------------------
>
>                 Key: CRAIL-98
>                 URL: https://issues.apache.org/jira/browse/CRAIL-98
>             Project: Apache Crail
>          Issue Type: Bug
>    Affects Versions: 1.2
>            Reporter: David Crespi
>            Assignee: Jonas Pfefferle
>            Priority: Major
>
> Running the latest Crail: v1.1-7-ga6e622f
> I’m running Crail as the temporary backend storage for Terasort.
> After each section (TeraGen, TeraSort, TeraVerify)
> the program waits until a Cntl-C is given, then moves on to the next
> section. 
>  
> Here’s a small snippet of the output.  Terasort waits where the
> bolded “Number of records” is listed, until the ^c is given.  Each of the 
> three programs does the same, but the program does finish without errors.
> 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:TaskSet_1.0, 
> runningTasks: 1
> 19/06/18 15:13:19 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) 
> in 142 ms on 192.168.3.10 (executor 4) (1/2)
> 19/06/18 15:13:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory 
> on 192.168.3.12:34011 (size: 1825.0 B, free: 366.3 MB)
> 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_1.0, 
> runningTasks: 0
> 19/06/18 15:13:19 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) 
> in 977 ms on 192.168.3.12 (executor 3) (2/2)
> 19/06/18 15:13:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks 
> have all completed, from pool
> 19/06/18 15:13:19 INFO DAGScheduler: ResultStage 1 (count at 
> TeraGen.scala:94) finished in 0.995 s
> 19/06/18 15:13:19 DEBUG DAGScheduler: After removal of stage 1, remaining 
> stages = 0
> 19/06/18 15:13:19 INFO DAGScheduler: Job 1 finished: count at 
> TeraGen.scala:94, took 1.003537 s
> *Number of records written: 10000*
> {color:#FF0000}*^C*{color}19/06/18 15:13:36 INFO SparkContext: Invoking 
> stop() from shutdown hook



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CRAIL-98) Crail, used as temporary results storage (type2 storage), misses close from TeraSort

Reply via email to