[jira] [Commented] (SPARK-2704) ConnectionManager threads should be named and daemon

[Apache Spark (JIRA)]

    [ 
https://issues.apache.org/jira/browse/SPARK-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075478#comment-14075478
 ] 

Apache Spark commented on SPARK-2704:
-------------------------------------

User 'rxin' has created a pull request for this issue:
https://github.com/apache/spark/pull/1604

> ConnectionManager threads should be named and daemon
> ----------------------------------------------------
>
>                 Key: SPARK-2704
>                 URL: https://issues.apache.org/jira/browse/SPARK-2704
>             Project: Spark
>          Issue Type: Bug
>            Reporter: Reynold Xin
>            Assignee: Reynold Xin
>
> handleMessageExecutor, handleReadWriteExecutor, and handleConnectExecutor are 
> not marked as daemon and not named. I think there exists some condition in 
> which Spark programs won't terminate because of this.
> Stack dumped (look at pool-1-thread-3, pool-2-thread-1, etc)
> {code}
> 2014-07-26 10:53:59
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode):
> "Attach Listener" daemon prio=5 tid=0x00007fc03480d800 nid=0x16303 waiting on 
> condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "ForkJoinPool-5-worker-15" daemon prio=5 tid=0x00007fc032b6f800 nid=0x5713 
> waiting on condition [0x000000011bf69000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007f5b5de28> (a 
> scala.concurrent.forkjoin.ForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker2-akka.actor.default-dispatcher-15" daemon prio=5 
> tid=0x00007fc031a9c800 nid=0x580b waiting on condition [0x000000011be66000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb00c88> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker2-akka.actor.default-dispatcher-14" daemon prio=5 
> tid=0x00007fc0358d3800 nid=0x3b17 waiting on condition [0x00000001111c9000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb00c88> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "DestroyJavaVM" prio=5 tid=0x00007fc031bc7000 nid=0xd17 waiting on condition 
> [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Result resolver thread-3" daemon prio=5 tid=0x00007fc0350e3000 nid=0x16003 
> waiting on condition [0x000000011bd63000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb10198> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "Result resolver thread-2" daemon prio=5 tid=0x00007fc031a9b800 nid=0x15e03 
> waiting on condition [0x000000011bc60000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb10198> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "Result resolver thread-1" daemon prio=5 tid=0x00007fc034aa4000 nid=0x15c03 
> waiting on condition [0x000000011bb5d000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb10198> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "Result resolver thread-0" daemon prio=5 tid=0x00007fc034aa0800 nid=0x15a03 
> waiting on condition [0x000000011ba5a000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb10198> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "pool-1-thread-4" prio=5 tid=0x00007fc032254000 nid=0x15803 waiting on 
> condition [0x000000011b957000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44098> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>       at 
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "pool-1-thread-3" prio=5 tid=0x00007fc032253000 nid=0x15603 waiting on 
> condition [0x000000011b854000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44098> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>       at 
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "pool-2-thread-4" prio=5 tid=0x00007fc03590a000 nid=0x15403 waiting on 
> condition [0x000000011b751000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb10300> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>       at 
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "pool-2-thread-3" prio=5 tid=0x00007fc035909800 nid=0x15203 waiting on 
> condition [0x000000011b64e000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb10300> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>       at 
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "pool-3-thread-1" prio=5 tid=0x00007fc031a9b000 nid=0x15003 waiting on 
> condition [0x000000011b54b000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb54198> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>       at 
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "pool-1-thread-2" prio=5 tid=0x00007fc034a9e800 nid=0x14e03 waiting on 
> condition [0x000000011b448000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44098> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>       at 
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "pool-2-thread-2" prio=5 tid=0x00007fc034a95800 nid=0x14c03 waiting on 
> condition [0x000000011b345000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb10300> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>       at 
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "pool-1-thread-1" prio=5 tid=0x00007fc032a79800 nid=0x14a03 waiting on 
> condition [0x000000011b242000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44098> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>       at 
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "pool-2-thread-1" prio=5 tid=0x00007fc031bc3000 nid=0x14803 waiting on 
> condition [0x000000011b13f000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb10300> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489)
>       at 
> java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "spark-akka.actor.default-dispatcher-16" daemon prio=5 tid=0x00007fc0321a2800 
> nid=0x14603 waiting on condition [0x000000011b03c000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44260> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "spark-akka.actor.default-dispatcher-15" daemon prio=5 tid=0x00007fc0321a1800 
> nid=0x14403 waiting on condition [0x000000011af39000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44260> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "File appending thread for 
> /scratch/rxin/spark-1/work/app-20140726103835-0000/1/stderr" daemon prio=5 
> tid=0x00007fc0321a8000 nid=0x13c07 runnable [0x000000011ad33000]
>    java.lang.Thread.State: RUNNABLE
>       at java.io.FileInputStream.readBytes(Native Method)
>       at java.io.FileInputStream.read(FileInputStream.java:272)
>       at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>       at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>       - locked <0x00000007feb2a238> (a 
> java.lang.UNIXProcess$ProcessPipeInputStream)
>       at java.io.FilterInputStream.read(FilterInputStream.java:107)
>       at 
> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>       at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1218)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
> "File appending thread for 
> /scratch/rxin/spark-1/work/app-20140726103835-0000/1/stdout" daemon prio=5 
> tid=0x00007fc0350b5000 nid=0x14007 runnable [0x000000011ab2d000]
>    java.lang.Thread.State: RUNNABLE
>       at java.io.FileInputStream.readBytes(Native Method)
>       at java.io.FileInputStream.read(FileInputStream.java:272)
>       at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>       at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>       - locked <0x00000007feb12b60> (a 
> java.lang.UNIXProcess$ProcessPipeInputStream)
>       at java.io.FilterInputStream.read(FilterInputStream.java:107)
>       at 
> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>       at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1218)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
> "File appending thread for 
> /scratch/rxin/spark-1/work/app-20140726103835-0000/0/stderr" daemon prio=5 
> tid=0x00007fc034102000 nid=0x14307 runnable [0x000000011ae36000]
>    java.lang.Thread.State: RUNNABLE
>       at java.io.FileInputStream.readBytes(Native Method)
>       at java.io.FileInputStream.read(FileInputStream.java:272)
>       at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>       at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>       - locked <0x00000007feb46478> (a 
> java.lang.UNIXProcess$ProcessPipeInputStream)
>       at java.io.FilterInputStream.read(FilterInputStream.java:107)
>       at 
> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>       at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1218)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
> "File appending thread for 
> /scratch/rxin/spark-1/work/app-20140726103835-0000/0/stdout" daemon prio=5 
> tid=0x00007fc034a9b800 nid=0x14207 runnable [0x000000011ac30000]
>    java.lang.Thread.State: RUNNABLE
>       at java.io.FileInputStream.readBytes(Native Method)
>       at java.io.FileInputStream.read(FileInputStream.java:272)
>       at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>       at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>       - locked <0x00000007feb221e8> (a 
> java.lang.UNIXProcess$ProcessPipeInputStream)
>       at java.io.FilterInputStream.read(FilterInputStream.java:107)
>       at 
> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>       at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1218)
>       at 
> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
> "process reaper" daemon prio=5 tid=0x00007fc0321a0800 nid=0x13a03 runnable 
> [0x000000011201b000]
>    java.lang.Thread.State: RUNNABLE
>       at java.lang.UNIXProcess.waitForProcessExit(Native Method)
>       at java.lang.UNIXProcess.access$200(UNIXProcess.java:54)
>       at java.lang.UNIXProcess$3.run(UNIXProcess.java:174)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "process reaper" daemon prio=5 tid=0x00007fc034171800 nid=0x13803 runnable 
> [0x0000000111fc4000]
>    java.lang.Thread.State: RUNNABLE
>       at java.lang.UNIXProcess.waitForProcessExit(Native Method)
>       at java.lang.UNIXProcess.access$200(UNIXProcess.java:54)
>       at java.lang.UNIXProcess$3.run(UNIXProcess.java:174)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "ExecutorRunner for app-20140726103835-0000/0" daemon prio=5 
> tid=0x00007fc03592a000 nid=0x13603 in Object.wait() [0x000000011aa2a000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007feb38140> (a java.lang.UNIXProcess)
>       at java.lang.Object.wait(Object.java:503)
>       at java.lang.UNIXProcess.waitFor(UNIXProcess.java:210)
>       - locked <0x00000007feb38140> (a java.lang.UNIXProcess)
>       at 
> org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:160)
>       at 
> org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:62)
> "ExecutorRunner for app-20140726103835-0000/1" daemon prio=5 
> tid=0x00007fc031b89800 nid=0x13403 in Object.wait() [0x000000011a927000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007feb544b0> (a java.lang.UNIXProcess)
>       at java.lang.Object.wait(Object.java:503)
>       at java.lang.UNIXProcess.waitFor(UNIXProcess.java:210)
>       - locked <0x00000007feb544b0> (a java.lang.UNIXProcess)
>       at 
> org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:160)
>       at 
> org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:62)
> "spark-akka.actor.default-dispatcher-14" daemon prio=5 tid=0x00007fc035931800 
> nid=0x13203 waiting on condition [0x000000011a824000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44260> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkMaster-akka.actor.default-dispatcher-17" daemon prio=5 
> tid=0x00007fc0330e3000 nid=0x13003 waiting on condition [0x000000011a721000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb22518> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "spark-akka.actor.default-dispatcher-13" daemon prio=5 tid=0x00007fc0330e2000 
> nid=0x12e03 waiting on condition [0x000000011a61e000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44260> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkMaster-akka.actor.default-dispatcher-16" daemon prio=5 
> tid=0x00007fc034a45000 nid=0x12c03 waiting on condition [0x000000011a51b000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb22518> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker1-akka.actor.default-dispatcher-14" daemon prio=5 
> tid=0x00007fc031b81800 nid=0x12a03 waiting on condition [0x000000011a418000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb226e8> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkMaster-akka.actor.default-dispatcher-15" daemon prio=5 
> tid=0x00007fc0350b2800 nid=0x12803 waiting on condition [0x000000011a315000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb22518> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker1-akka.actor.default-dispatcher-13" daemon prio=5 
> tid=0x00007fc035931000 nid=0x12603 waiting on condition [0x000000011a212000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb226e8> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "Hashed wheel timer #4" daemon prio=5 tid=0x00007fc03592d800 nid=0x12403 
> waiting on condition [0x000000011a10f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at 
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:503)
>       at 
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:401)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at java.lang.Thread.run(Thread.java:744)
> "Hashed wheel timer #3" daemon prio=5 tid=0x00007fc03414e000 nid=0x12203 
> waiting on condition [0x000000011a00c000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at 
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:503)
>       at 
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:401)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at java.lang.Thread.run(Thread.java:744)
> "Hashed wheel timer #1" daemon prio=5 tid=0x00007fc03592d000 nid=0x12003 
> waiting on condition [0x0000000119f09000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at 
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:503)
>       at 
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:401)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at java.lang.Thread.run(Thread.java:744)
> "Spark Context Cleaner" daemon prio=5 tid=0x00007fc034a3e000 nid=0x11e03 in 
> Object.wait() [0x0000000119e06000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007feb38390> (a java.lang.ref.ReferenceQueue$Lock)
>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>       - locked <0x00000007feb38390> (a java.lang.ref.ReferenceQueue$Lock)
>       at 
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:117)
>       at 
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:115)
>       at 
> org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply(ContextCleaner.scala:115)
>       at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1218)
>       at 
> org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:114)
>       at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:65)
> "qtp1722016480-116" daemon prio=5 tid=0x00007fc034160800 nid=0x11c03 waiting 
> on condition [0x0000000119d03000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb2a688> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1722016480-115" daemon prio=5 tid=0x00007fc0322c6000 nid=0x11a03 waiting 
> on condition [0x0000000119c00000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb2a688> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1722016480-114" daemon prio=5 tid=0x00007fc0358ff000 nid=0x11803 waiting 
> on condition [0x0000000119afd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb2a688> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1722016480-113" daemon prio=5 tid=0x00007fc0322bb000 nid=0x11603 waiting 
> on condition [0x00000001199fa000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb2a688> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1722016480-112 Acceptor1 SelectChannelConnector@0.0.0.0:50373" daemon 
> prio=5 tid=0x00007fc0322ba000 nid=0x11403 runnable [0x00000001198f7000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>       at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
>       - locked <0x00000007febf63b8> (a java.lang.Object)
>       at 
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>       at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1722016480-111 Acceptor0 SelectChannelConnector@0.0.0.0:50373" daemon 
> prio=5 tid=0x00007fc03415f800 nid=0x11203 waiting for monitor entry 
> [0x00000001197f4000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>       at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
>       - waiting to lock <0x00000007febf63b8> (a java.lang.Object)
>       at 
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>       at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1722016480-110 Selector1" daemon prio=5 tid=0x00007fc0350b0800 
> nid=0x11003 runnable [0x00000001196f1000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb2aeb0> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb2aea0> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb2ad70> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1722016480-109 Selector0" daemon prio=5 tid=0x00007fc034a3c800 
> nid=0x10e03 runnable [0x00000001195ee000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb2ab68> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb2ab58> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb2aa18> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O server boss #24" daemon prio=5 tid=0x00007fc03415f000 nid=0x10c03 
> runnable [0x00000001194eb000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb38b28> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb38b38> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb38ad8> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>       at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #23" daemon prio=5 tid=0x00007fc032a50000 nid=0x10a03 
> runnable [0x00000001193e8000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb2b4c0> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb2b4d0> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb2b470> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #22" daemon prio=5 tid=0x00007fc034a3b000 nid=0x10803 
> runnable [0x00000001192e5000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb244b8> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb244c8> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb2b978> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O boss #21" daemon prio=5 tid=0x00007fc0329f1800 nid=0x10603 runnable 
> [0x00000001191e2000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb3b328> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb3b318> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb3b0d8> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #20" daemon prio=5 tid=0x00007fc0329f1000 nid=0x10403 
> runnable [0x00000001190df000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb24780> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb24790> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb24730> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #19" daemon prio=5 tid=0x00007fc0329ee000 nid=0x10203 
> runnable [0x0000000118fdc000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb25018> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb25008> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb24c10> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1769933516-100" daemon prio=5 tid=0x00007fc0358fe000 nid=0x10003 waiting 
> on condition [0x0000000118ed9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb25850> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1769933516-99" daemon prio=5 tid=0x00007fc0358fd800 nid=0xfe03 waiting on 
> condition [0x0000000118dd6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb25850> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1769933516-98" daemon prio=5 tid=0x00007fc03220b800 nid=0xfc03 waiting on 
> condition [0x0000000118cd3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb25850> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1769933516-97" daemon prio=5 tid=0x00007fc03220b000 nid=0xfa03 waiting on 
> condition [0x0000000118bd0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb25850> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1769933516-96 Acceptor1 SelectChannelConnector@0.0.0.0:50371" daemon 
> prio=5 tid=0x00007fc03220a000 nid=0xf803 runnable [0x0000000118acd000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>       at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
>       - locked <0x00000007feb264a8> (a java.lang.Object)
>       at 
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>       at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1769933516-95 Acceptor0 SelectChannelConnector@0.0.0.0:50371" daemon 
> prio=5 tid=0x00007fc032209800 nid=0xf603 waiting for monitor entry 
> [0x00000001189ca000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>       at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
>       - waiting to lock <0x00000007feb264a8> (a java.lang.Object)
>       at 
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>       at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1769933516-94 Selector1" daemon prio=5 tid=0x00007fc032208800 nid=0xf403 
> runnable [0x00000001188c7000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb26018> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb26008> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb25ed8> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1769933516-93 Selector0" daemon prio=5 tid=0x00007fc031b8b000 nid=0xf203 
> runnable [0x00000001187c4000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feb25cd0> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feb25cc0> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feb25b90> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "sparkWorker2-akka.actor.default-dispatcher-6" daemon prio=5 
> tid=0x00007fc03414f800 nid=0xf003 waiting on condition [0x00000001186c1000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb00c88> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker2-akka.actor.default-dispatcher-5" daemon prio=5 
> tid=0x00007fc0349d0000 nid=0xee03 waiting on condition [0x00000001185be000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb00c88> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker2-akka.actor.default-dispatcher-4" daemon prio=5 
> tid=0x00007fc031ba8800 nid=0xec03 waiting on condition [0x00000001184bb000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb00c88> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker2-akka.actor.default-dispatcher-3" daemon prio=5 
> tid=0x00007fc031ba7800 nid=0xea03 waiting on condition [0x00000001183b8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb00c88> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker2-akka.actor.default-dispatcher-2" daemon prio=5 
> tid=0x00007fc0349cf800 nid=0xe803 waiting on condition [0x00000001182b5000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb00c88> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker2-scheduler-1" daemon prio=5 tid=0x00007fc0349df800 nid=0xe603 
> waiting on condition [0x00000001181b2000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>       at 
> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:393)
>       at 
> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O server boss #18" daemon prio=5 tid=0x00007fc0329eb800 nid=0xe403 
> runnable [0x00000001180af000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed3bb20> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed3bb30> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed3bad0> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>       at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #17" daemon prio=5 tid=0x00007fc032212800 nid=0xe203 runnable 
> [0x0000000117fac000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed94608> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed94618> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed945b8> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #16" daemon prio=5 tid=0x00007fc03227f000 nid=0xe003 runnable 
> [0x0000000117ea9000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed949e0> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed949f0> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fec5d230> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O boss #15" daemon prio=5 tid=0x00007fc03227e800 nid=0xde03 runnable 
> [0x0000000117da6000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fec5f068> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fec5f058> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fec5ee18> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #14" daemon prio=5 tid=0x00007fc0329cf000 nid=0xdc03 runnable 
> [0x0000000117ca3000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed94d78> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed94d88> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed94d28> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #13" daemon prio=5 tid=0x00007fc0329cc800 nid=0xda03 runnable 
> [0x0000000117ba0000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed611c0> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed611d0> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed61170> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "sparkWorker1-akka.actor.default-dispatcher-5" daemon prio=5 
> tid=0x00007fc0329d5000 nid=0xd803 waiting on condition [0x0000000117a9d000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb226e8> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker1-akka.actor.default-dispatcher-4" daemon prio=5 
> tid=0x00007fc031ac8000 nid=0xd603 waiting on condition [0x000000011799a000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb226e8> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker1-akka.actor.default-dispatcher-3" daemon prio=5 
> tid=0x00007fc0329d4000 nid=0xd403 waiting on condition [0x0000000117897000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb226e8> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker1-akka.actor.default-dispatcher-2" daemon prio=5 
> tid=0x00007fc0329d2800 nid=0xd203 waiting on condition [0x0000000117794000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb226e8> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkWorker1-scheduler-1" daemon prio=5 tid=0x00007fc0329d2000 nid=0xd003 
> waiting on condition [0x0000000117691000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>       at 
> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:393)
>       at 
> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1858873692-73" daemon prio=5 tid=0x00007fc0340a4000 nid=0xce03 waiting on 
> condition [0x000000011758e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fed61af8> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1858873692-72" daemon prio=5 tid=0x00007fc0329a6800 nid=0xcc03 waiting on 
> condition [0x000000011748b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fed61af8> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1858873692-71" daemon prio=5 tid=0x00007fc0358fc800 nid=0xca03 waiting on 
> condition [0x0000000117388000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fed61af8> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1858873692-70" daemon prio=5 tid=0x00007fc031ba7000 nid=0xc803 waiting on 
> condition [0x0000000117285000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fed61af8> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1858873692-69 Acceptor1 SelectChannelConnector@0.0.0.0:50369" daemon 
> prio=5 tid=0x00007fc0358fc000 nid=0xc603 waiting for monitor entry 
> [0x0000000117182000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>       at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
>       - waiting to lock <0x00000007fed95600> (a java.lang.Object)
>       at 
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>       at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1858873692-68 Acceptor0 SelectChannelConnector@0.0.0.0:50369" daemon 
> prio=5 tid=0x00007fc031ba6000 nid=0xc403 runnable [0x000000011707f000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>       at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
>       - locked <0x00000007fed95600> (a java.lang.Object)
>       at 
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>       at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1858873692-67 Selector1" daemon prio=5 tid=0x00007fc031ba5800 nid=0xc203 
> runnable [0x0000000116f7c000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed1ce30> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed1ce40> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed1cde0> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp1858873692-66 Selector0" daemon prio=5 tid=0x00007fc031ac9800 nid=0xc003 
> runnable [0x0000000116e79000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed61f78> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed61f68> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed61e38> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O server boss #12" daemon prio=5 tid=0x00007fc0329a3800 nid=0xbe03 
> runnable [0x0000000116d76000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed1d608> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed1d618> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed1d5b8> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>       at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #11" daemon prio=5 tid=0x00007fc0329a3000 nid=0xbc03 runnable 
> [0x0000000116c73000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed62408> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed62418> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed623b8> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #10" daemon prio=5 tid=0x00007fc03227d800 nid=0xba03 runnable 
> [0x0000000116b70000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fedb21d8> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fedb21e8> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed62948> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O boss #9" daemon prio=5 tid=0x00007fc032225000 nid=0xb803 runnable 
> [0x0000000116a6d000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fedb23e0> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fedb23f0> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fedb2390> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #8" daemon prio=5 tid=0x00007fc032224800 nid=0xb603 runnable 
> [0x000000011696a000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fedf1110> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fedf1120> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fedf10c0> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #7" daemon prio=5 tid=0x00007fc0330e7000 nid=0xb403 runnable 
> [0x0000000116867000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fedf1660> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fedf1650> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fedf1530> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "sparkMaster-akka.actor.default-dispatcher-7" daemon prio=5 
> tid=0x00007fc03419b000 nid=0xb203 waiting on condition [0x0000000116764000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb22518> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkMaster-akka.actor.default-dispatcher-6" daemon prio=5 
> tid=0x00007fc0329f6800 nid=0xb003 waiting on condition [0x0000000116661000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb22518> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkMaster-akka.actor.default-dispatcher-5" daemon prio=5 
> tid=0x00007fc0328eb000 nid=0xae03 waiting on condition [0x000000011655e000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb22518> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkMaster-akka.actor.default-dispatcher-4" daemon prio=5 
> tid=0x00007fc031ac7000 nid=0xac03 waiting on condition [0x000000011645b000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb22518> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkMaster-akka.actor.default-dispatcher-3" daemon prio=5 
> tid=0x00007fc032221000 nid=0xaa03 waiting on condition [0x0000000116358000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb22518> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkMaster-akka.actor.default-dispatcher-2" daemon prio=5 
> tid=0x00007fc032220000 nid=0xa803 waiting on condition [0x0000000116255000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb22518> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkMaster-scheduler-1" daemon prio=5 tid=0x00007fc0349e3800 nid=0xa603 
> waiting on condition [0x0000000116152000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>       at 
> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:393)
>       at 
> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
>       at java.lang.Thread.run(Thread.java:744)
> "Timer-0" daemon prio=5 tid=0x00007fc0349d5800 nid=0xa403 in Object.wait() 
> [0x0000000114005000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007fedf2b70> (a java.util.TaskQueue)
>       at java.lang.Object.wait(Object.java:503)
>       at java.util.TimerThread.mainLoop(Timer.java:526)
>       - locked <0x00000007fedf2b70> (a java.util.TaskQueue)
>       at java.util.TimerThread.run(Timer.java:505)
> "SparkListenerBus" daemon prio=5 tid=0x00007fc032993000 nid=0xa207 waiting on 
> condition [0x0000000115e9c000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fedf2ed8> (a 
> java.util.concurrent.Semaphore$NonfairSync)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>       at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>       at 
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:48)
>       at 
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
>       at 
> org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply(LiveListenerBus.scala:47)
>       at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1218)
>       at 
> org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:46)
> "qtp205855311-49" daemon prio=5 tid=0x00007fc03598b800 nid=0x9e03 waiting on 
> condition [0x0000000114726000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fed80c10> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp205855311-48" daemon prio=5 tid=0x00007fc034a12800 nid=0x9c03 waiting on 
> condition [0x0000000114623000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fed80c10> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp205855311-47" daemon prio=5 tid=0x00007fc031a2f800 nid=0x9a03 waiting on 
> condition [0x0000000114520000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fed80c10> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp205855311-46" daemon prio=5 tid=0x00007fc0340ac800 nid=0x9803 waiting on 
> condition [0x000000011441d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fed80c10> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp205855311-45 Acceptor1 SelectChannelConnector@0.0.0.0:4040" daemon prio=5 
> tid=0x00007fc032992800 nid=0x9603 waiting for monitor entry 
> [0x000000011431a000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>       at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
>       - waiting to lock <0x00000007fed818a0> (a java.lang.Object)
>       at 
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>       at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp205855311-44 Acceptor0 SelectChannelConnector@0.0.0.0:4040" daemon prio=5 
> tid=0x00007fc03412f000 nid=0x9403 runnable [0x0000000114217000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>       at 
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
>       - locked <0x00000007fed818a0> (a java.lang.Object)
>       at 
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
>       at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp205855311-43 Selector1" daemon prio=5 tid=0x00007fc032991800 nid=0x9203 
> runnable [0x0000000114114000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed81560> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed81550> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed81420> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp205855311-42 Selector0" daemon prio=5 tid=0x00007fc032991000 nid=0x9003 
> runnable [0x0000000113f02000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fed811e0> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fed811d0> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fed810a0> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>       at 
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "SPARK_CONTEXT cleanup timer" daemon prio=5 tid=0x00007fc034124000 nid=0x8e03 
> in Object.wait() [0x0000000112774000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007fee79858> (a java.util.TaskQueue)
>       at java.lang.Object.wait(Object.java:503)
>       at java.util.TimerThread.mainLoop(Timer.java:526)
>       - locked <0x00000007fee79858> (a java.util.TaskQueue)
>       at java.util.TimerThread.run(Timer.java:505)
> "qtp2111378219-38" daemon prio=5 tid=0x00007fc034092000 nid=0x8c03 waiting on 
> condition [0x0000000113dff000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fee79be0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp2111378219-37" daemon prio=5 tid=0x00007fc032262000 nid=0x8a03 waiting on 
> condition [0x0000000113cfc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fee79be0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp2111378219-36" daemon prio=5 tid=0x00007fc03494f800 nid=0x8803 waiting on 
> condition [0x0000000113bf9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fee79be0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp2111378219-35" daemon prio=5 tid=0x00007fc03297e000 nid=0x8603 waiting on 
> condition [0x0000000113af6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fee79be0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp2111378219-34" daemon prio=5 tid=0x00007fc032261000 nid=0x8403 waiting on 
> condition [0x00000001139f3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fee79be0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp2111378219-33" daemon prio=5 tid=0x00007fc032260800 nid=0x8203 waiting on 
> condition [0x00000001138f0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fee79be0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp2111378219-32" daemon prio=5 tid=0x00007fc034939000 nid=0x8003 waiting on 
> condition [0x00000001137ed000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007fee79be0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>       at java.lang.Thread.run(Thread.java:744)
> "qtp2111378219-31 Acceptor0 SocketConnector@0.0.0.0:50367" daemon prio=5 
> tid=0x00007fc034895000 nid=0x7e03 runnable [0x00000001136ea000]
>    java.lang.Thread.State: RUNNABLE
>       at java.net.PlainSocketImpl.socketAccept(Native Method)
>       at 
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
>       at java.net.ServerSocket.implAccept(ServerSocket.java:530)
>       at java.net.ServerSocket.accept(ServerSocket.java:498)
>       at 
> org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
>       at 
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>       at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>       at java.lang.Thread.run(Thread.java:744)
> "Connection manager future execution context-0" daemon prio=5 
> tid=0x00007fc03587d800 nid=0x7c03 waiting on condition [0x00000001135e7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feeaea38> (a 
> java.util.concurrent.SynchronousQueue$TransferStack)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>       at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
>       at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "BROADCAST_VARS cleanup timer" daemon prio=5 tid=0x00007fc032257800 
> nid=0x7a03 in Object.wait() [0x00000001134e4000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007feeaf038> (a java.util.TaskQueue)
>       at java.lang.Object.wait(Object.java:503)
>       at java.util.TimerThread.mainLoop(Timer.java:526)
>       - locked <0x00000007feeaf038> (a java.util.TaskQueue)
>       at java.util.TimerThread.run(Timer.java:505)
> "BLOCK_MANAGER cleanup timer" daemon prio=5 tid=0x00007fc032256800 nid=0x7803 
> in Object.wait() [0x00000001133e1000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007fee0d090> (a java.util.TaskQueue)
>       at java.lang.Object.wait(Object.java:503)
>       at java.util.TimerThread.mainLoop(Timer.java:526)
>       - locked <0x00000007fee0d090> (a java.util.TaskQueue)
>       at java.util.TimerThread.run(Timer.java:505)
> "connection-manager-thread" daemon prio=5 tid=0x00007fc0319b4000 nid=0x7603 
> runnable [0x00000001132de000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007fee7a690> (a sun.nio.ch.Util$2)
>       - locked <0x00000007fee7a6a0> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fee7a640> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>       at 
> org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:303)
>       at 
> org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:116)
> "SHUFFLE_BLOCK_MANAGER cleanup timer" daemon prio=5 tid=0x00007fc0330ec000 
> nid=0x7403 in Object.wait() [0x00000001131db000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007feeaf3c8> (a java.util.TaskQueue)
>       at java.lang.Object.wait(Object.java:503)
>       at java.util.TimerThread.mainLoop(Timer.java:526)
>       - locked <0x00000007feeaf3c8> (a java.util.TaskQueue)
>       at java.util.TimerThread.run(Timer.java:505)
> "MAP_OUTPUT_TRACKER cleanup timer" daemon prio=5 tid=0x00007fc03504b800 
> nid=0x7203 in Object.wait() [0x00000001130d8000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007feee46b0> (a java.util.TaskQueue)
>       at java.lang.Object.wait(Object.java:503)
>       at java.util.TimerThread.mainLoop(Timer.java:526)
>       - locked <0x00000007feee46b0> (a java.util.TaskQueue)
>       at java.util.TimerThread.run(Timer.java:505)
> "New I/O server boss #6" daemon prio=5 tid=0x00007fc03197b800 nid=0x7003 
> runnable [0x0000000112fd5000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feee4ab8> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feee4ac8> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feee4a68> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
>       at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #5" daemon prio=5 tid=0x00007fc03198e800 nid=0x6e03 runnable 
> [0x0000000112ed2000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feeaf720> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feeaf730> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007fee92108> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #4" daemon prio=5 tid=0x00007fc03197a800 nid=0x6c03 runnable 
> [0x0000000112dcf000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feee5530> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feee5540> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feee54e0> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O boss #3" daemon prio=5 tid=0x00007fc031978000 nid=0x6a03 runnable 
> [0x0000000112ccc000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feebf658> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feebf648> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feebf408> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #2" daemon prio=5 tid=0x00007fc03587d000 nid=0x6803 runnable 
> [0x0000000112bc9000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feebfd08> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feebfcf8> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feebfbd8> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "New I/O worker #1" daemon prio=5 tid=0x00007fc032926800 nid=0x6603 runnable 
> [0x0000000112ac6000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>       at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>       at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000007feec04c8> (a sun.nio.ch.Util$2)
>       - locked <0x00000007feec04b8> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000007feec00c0> (a sun.nio.ch.KQueueSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
>       at 
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>       at 
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>       at 
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:744)
> "spark-akka.actor.default-dispatcher-5" daemon prio=5 tid=0x00007fc03191f000 
> nid=0x6403 waiting on condition [0x00000001129c3000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44260> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "spark-akka.actor.default-dispatcher-4" daemon prio=5 tid=0x00007fc031911800 
> nid=0x6203 waiting on condition [0x00000001128c0000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44260> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "spark-akka.actor.default-dispatcher-3" daemon prio=5 tid=0x00007fc0328e9000 
> nid=0x6003 waiting on condition [0x00000001125a9000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44260> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "spark-akka.actor.default-dispatcher-2" daemon prio=5 tid=0x00007fc032940000 
> nid=0x5e03 waiting on condition [0x00000001124a6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000007feb44260> (a 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
>       at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
>       at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>       at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "spark-scheduler-1" daemon prio=5 tid=0x00007fc032133000 nid=0x5c07 waiting 
> on condition [0x0000000112281000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
>       at 
> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:393)
>       at 
> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
>       at java.lang.Thread.run(Thread.java:744)
> "Service Thread" daemon prio=5 tid=0x00007fc035000000 nid=0x5303 runnable 
> [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" daemon prio=5 tid=0x00007fc034801000 nid=0x5103 waiting 
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" daemon prio=5 tid=0x00007fc031810800 nid=0x4f03 waiting 
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=5 tid=0x00007fc031810000 nid=0x4d03 runnable 
> [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=5 tid=0x00007fc03180f000 nid=0x3903 in Object.wait() 
> [0x0000000111098000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007feee6f30> (a java.lang.ref.ReferenceQueue$Lock)
>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>       - locked <0x00000007feee6f30> (a java.lang.ref.ReferenceQueue$Lock)
>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>       at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
> "Reference Handler" daemon prio=5 tid=0x00007fc032013000 nid=0x3703 in 
> Object.wait() [0x0000000110f95000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000007fee0dac0> (a java.lang.ref.Reference$Lock)
>       at java.lang.Object.wait(Object.java:503)
>       at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>       - locked <0x00000007fee0dac0> (a java.lang.ref.Reference$Lock)
> "VM Thread" prio=5 tid=0x00007fc03283e800 nid=0x3503 runnable 
> "GC task thread#0 (ParallelGC)" prio=5 tid=0x00007fc03200f800 nid=0x11f 
> runnable 
> "GC task thread#1 (ParallelGC)" prio=5 tid=0x00007fc032010000 nid=0x252b 
> runnable 
> "GC task thread#2 (ParallelGC)" prio=5 tid=0x00007fc032010800 nid=0x2903 
> runnable 
> "GC task thread#3 (ParallelGC)" prio=5 tid=0x00007fc032011800 nid=0x2b03 
> runnable 
> "GC task thread#4 (ParallelGC)" prio=5 tid=0x00007fc032012000 nid=0x2d03 
> runnable 
> "GC task thread#5 (ParallelGC)" prio=5 tid=0x00007fc032012800 nid=0x2f03 
> runnable 
> "GC task thread#6 (ParallelGC)" prio=5 tid=0x00007fc03180c000 nid=0x3103 
> runnable 
> "GC task thread#7 (ParallelGC)" prio=5 tid=0x00007fc03180c800 nid=0x3303 
> runnable 
> "VM Periodic Task Thread" prio=5 tid=0x00007fc032846800 nid=0x5503 waiting on 
> condition 
> JNI global references: 322
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)