[ https://issues.apache.org/jira/browse/SPARK-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Miner updated SPARK-18343: ------------------------------- Comment: was deleted (was: Any suggestions on how one might hunt down that library? I've included my build.sbt.) > FileSystem$Statistics$StatisticsDataReferenceCleaner hangs on s3 write > ---------------------------------------------------------------------- > > Key: SPARK-18343 > URL: https://issues.apache.org/jira/browse/SPARK-18343 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.0.1 > Environment: Spark 2.0.1 > Hadoop 2.7.1 > Mesos 1.0.1 > Ubuntu 14.04 > Reporter: Luke Miner > > I have a driver program where I write read data in from Cassandra using > spark, perform some operations, and then write out to JSON on S3. The program > runs fine when I use Spark 1.6.1 and the spark-cassandra-connector 1.6.0-M1. > However, if I try to upgrade to Spark 2.0.1 (hadoop 2.7.1) and > spark-cassandra-connector 2.0.0-M3, the program completes in the sense that > all the expected files are written to S3, but the program never terminates. > I do run `sc.stop()` at the end of the program. I am also using Mesos 1.0.1. > In both cases I use the default output committer. > From the thread dump (included below) it seems like it could be waiting on: > `org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner` > Code snippet: > {code} > // get MongoDB oplog operations > val operations = sc.cassandraTable[JsonOperation](keyspace, namespace) > .where("ts >= ? AND ts < ?", minTimestamp, maxTimestamp) > > // replay oplog operations into documents > val documents = operations > .spanBy(op => op.id) > .map { case (id: String, ops: Iterable[T]) => (id, apply(ops)) } > .filter { case (id, result) => result.isInstanceOf[Document] } > .map { case (id, document) => MergedDocument(id = id, document = > document > .asInstanceOf[Document]) > } > > // write documents to json on s3 > documents > .map(document => document.toJson) > .coalesce(partitions) > .saveAsTextFile(path, classOf[GzipCodec]) > sc.stop() > {code} > build.sbt > {code} > scalaVersion := "2.11.8" > ivyScala := ivyScala.value map { > _.copy(overrideScalaVersion = true) > } > resolvers += Resolver.mavenLocal > // spark > libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0" % > "provided" > libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0" % > "provided" > // spark cassandra connector > libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % > "2.0.0-M3" excludeAll > ExclusionRule(organization = "org.apache.spark") > // other libraries > libraryDependencies += "org.json4s" %% "json4s-native" % "3.2.11" > libraryDependencies += "org.rogach" %% "scallop" % "2.0.1" > libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "1.8.0" > // test > libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.6" % "test" > libraryDependencies += "junit" % "junit" % "4.11" % "test" > libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test" > assemblyOption in assembly := (assemblyOption in > assembly).value.copy(includeScala = false) > net.virtualvoid.sbt.graph.Plugin.graphSettings > fork := true > assemblyShadeRules in assembly := Seq( > ShadeRule.rename("com.google.common.**" -> "shadedguava.@1").inAll > ) > > assemblyMergeStrategy in assembly := { > case PathList(ps@_*) if ps.last endsWith ".properties" => > MergeStrategy.first > case x => > val oldStrategy = (assemblyMergeStrategy in assembly).value > oldStrategy(x) > } > scalacOptions += "-deprecation" > {code} > Thread dump on the driver: > {code} > 60 context-cleaner-periodic-gc TIMED_WAITING > 46 dag-scheduler-event-loop WAITING > 4389 DestroyJavaVM RUNNABLE > 12 dispatcher-event-loop-0 WAITING > 13 dispatcher-event-loop-1 WAITING > 14 dispatcher-event-loop-2 WAITING > 15 dispatcher-event-loop-3 WAITING > 47 driver-revive-thread TIMED_WAITING > 3 Finalizer WAITING > 82 ForkJoinPool-1-worker-17 WAITING > 43 heartbeat-receiver-event-loop-thread TIMED_WAITING > 93 java-sdk-http-connection-reaper TIMED_WAITING > 4387 java-sdk-progress-listener-callback-thread WAITING > 25 map-output-dispatcher-0 WAITING > 26 map-output-dispatcher-1 WAITING > 27 map-output-dispatcher-2 WAITING > 28 map-output-dispatcher-3 WAITING > 29 map-output-dispatcher-4 WAITING > 30 map-output-dispatcher-5 WAITING > 31 map-output-dispatcher-6 WAITING > 32 map-output-dispatcher-7 WAITING > 48 MesosCoarseGrainedSchedulerBackend-mesos-driver RUNNABLE > 44 netty-rpc-env-timeout TIMED_WAITING > 92 > org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner > WAITING > 62 pool-19-thread-1 TIMED_WAITING > 2 Reference Handler WAITING > 61 Scheduler-1112394071 TIMED_WAITING > 20 shuffle-server-0 RUNNABLE > 55 shuffle-server-0 RUNNABLE > 21 shuffle-server-1 RUNNABLE > 56 shuffle-server-1 RUNNABLE > 22 shuffle-server-2 RUNNABLE > 57 shuffle-server-2 RUNNABLE > 23 shuffle-server-3 RUNNABLE > 58 shuffle-server-3 RUNNABLE > 4 Signal Dispatcher RUNNABLE > 59 Spark Context Cleaner TIMED_WAITING > 9 SparkListenerBus WAITING > 35 SparkUI-35-selector-ServerConnectorManager@651d3734/0 RUNNABLE > 36 > SparkUI-36-acceptor-0@467924cb-ServerConnector@3b5eaf92{HTTP/1.1}{0.0.0.0:4040} > RUNNABLE > 37 SparkUI-37-selector-ServerConnectorManager@651d3734/1 RUNNABLE > 38 SparkUI-38 TIMED_WAITING > 39 SparkUI-39 TIMED_WAITING > 40 SparkUI-40 TIMED_WAITING > 41 SparkUI-41 RUNNABLE > 42 SparkUI-42 TIMED_WAITING > 438 task-result-getter-0 WAITING > 450 task-result-getter-1 WAITING > 489 task-result-getter-2 WAITING > 492 task-result-getter-3 WAITING > 75 threadDeathWatcher-2-1 TIMED_WAITING > 45 Timer-0 WAITING > {code} > Thread dump on the executors. It's the same on all of them: > {code} > 24 dispatcher-event-loop-0 WAITING > 25 dispatcher-event-loop-1 WAITING > 26 dispatcher-event-loop-2 RUNNABLE > 27 dispatcher-event-loop-3 WAITING > 39 driver-heartbeater TIMED_WAITING > 3 Finalizer WAITING > 58 java-sdk-http-connection-reaper TIMED_WAITING > 75 java-sdk-progress-listener-callback-thread WAITING > 1 main TIMED_WAITING > 33 netty-rpc-env-timeout TIMED_WAITING > 55 > org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner > WAITING > 59 pool-17-thread-1 TIMED_WAITING > 2 Reference Handler WAITING > 28 shuffle-client-0 RUNNABLE > 35 shuffle-client-0 RUNNABLE > 41 shuffle-client-0 RUNNABLE > 37 shuffle-server-0 RUNNABLE > 5 Signal Dispatcher RUNNABLE > 23 threadDeathWatcher-2-1 TIMED_WAITING > {code} > Jstack of an executor: > {code} > ubuntu@ip-10-0-230-88:~$ sudo jstack 21811 > 2016-11-08 21:38:02 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode): > "Attach Listener" daemon prio=10 tid=0x00007f8234003800 nid=0x5a4c waiting on > condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "java-sdk-progress-listener-callback-thread" daemon prio=10 > tid=0x00007f8218001000 nid=0x55c5 waiting on condition [0x00007f81e98d5000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000000078797f4f8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "pool-17-thread-1" daemon prio=10 tid=0x00007f82141f9000 nid=0x5597 waiting > on condition [0x00007f81fc2bb000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000000074d9008e8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "java-sdk-http-connection-reaper" daemon prio=10 tid=0x00007f820837e000 > nid=0x5596 waiting on condition [0x00007f81fc3bc000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:112) > "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" > daemon prio=10 tid=0x00007f8208352800 nid=0x5594 in Object.wait() > [0x00007f824cc13000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x0000000756803100> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x0000000756803100> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at > org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063) > at java.lang.Thread.run(Thread.java:745) > "shuffle-client-0" daemon prio=10 tid=0x00007f8208110800 nid=0x5593 runnable > [0x00007f824ca11000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x0000000756803238> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x0000000756803258> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007568031f0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-client-0" daemon prio=10 tid=0x00007f820803b800 nid=0x5578 runnable > [0x00007f824c704000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007568033e0> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x0000000756803400> (a java.util.Collections$UnmodifiableSet) > - locked <0x0000000756803398> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "driver-heartbeater" daemon prio=10 tid=0x00007f8200047800 nid=0x5573 waiting > on condition [0x00007f81fdefb000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007568036b8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-0" daemon prio=10 tid=0x00007f8200044000 nid=0x5572 runnable > [0x00007f81fdffc000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007568038c0> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000007568038e0> (a java.util.Collections$UnmodifiableSet) > - locked <0x0000000756803878> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-client-0" daemon prio=10 tid=0x000000000222c000 nid=0x5571 runnable > [0x00007f824c1ff000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x0000000756803a68> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x0000000756803a88> (a java.util.Collections$UnmodifiableSet) > - locked <0x0000000756803a20> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "netty-rpc-env-timeout" daemon prio=10 tid=0x00007f8285248000 nid=0x5570 > waiting on condition [0x00007f824c300000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000756803b80> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-3" daemon prio=10 tid=0x00007f82851f4800 nid=0x556e > waiting on condition [0x00007f824c502000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000756802418> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-2" daemon prio=10 tid=0x00007f82851f3800 nid=0x556d > waiting on condition [0x00007f824c805000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000756802418> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-1" daemon prio=10 tid=0x00007f82851f3000 nid=0x556c > waiting on condition [0x00007f824cf15000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000756802418> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-0" daemon prio=10 tid=0x00007f82851f2000 nid=0x556b > waiting on condition [0x00007f824c906000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000756802418> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "threadDeathWatcher-2-1" daemon prio=10 tid=0x00007f820400e000 nid=0x5567 > waiting on condition [0x00007f824c603000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" daemon prio=10 tid=0x00007f82842ae000 nid=0x555a runnable > [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=10 tid=0x00007f82842ab000 nid=0x5559 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=10 tid=0x00007f82842a9000 nid=0x5558 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=10 tid=0x00007f82842a6800 nid=0x5557 runnable > [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 > tid=0x00007f82842a4800 nid=0x5556 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=10 tid=0x00007f8284282800 nid=0x5555 in Object.wait() > [0x00007f8280dfc000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007568040a0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007568040a0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=10 tid=0x00007f8284280800 nid=0x5554 in > Object.wait() [0x00007f8280efd000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007568040e0> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000007568040e0> (a java.lang.ref.Reference$Lock) > "main" prio=10 tid=0x00007f8284021000 nid=0x5547 waiting on condition > [0x00007f828da05000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x0000000756804ac8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468) > at > org.apache.spark.rpc.netty.Dispatcher.awaitTermination(Dispatcher.scala:180) > at > org.apache.spark.rpc.netty.NettyRpcEnv.awaitTermination(NettyRpcEnv.scala:273) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:217) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:71) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:70) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:174) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:270) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala) > "VM Thread" prio=10 tid=0x00007f828427c000 nid=0x5553 runnable > "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f8284035800 > nid=0x5548 runnable > "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f8284037800 > nid=0x5549 runnable > "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f8284039000 > nid=0x554a runnable > "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f828403b000 > nid=0x554b runnable > "G1 Main Concurrent Mark GC Thread" prio=10 tid=0x00007f828404f800 nid=0x5551 > runnable > "Gang worker#0 (G1 Parallel Marking Threads)" prio=10 tid=0x00007f8284062000 > nid=0x5552 runnable > "G1 Concurrent Refinement Thread#0" prio=10 tid=0x00007f8284045800 nid=0x5550 > runnable > "G1 Concurrent Refinement Thread#1" prio=10 tid=0x00007f8284043800 nid=0x554f > runnable > "G1 Concurrent Refinement Thread#2" prio=10 tid=0x00007f8284041800 nid=0x554e > runnable > "G1 Concurrent Refinement Thread#3" prio=10 tid=0x00007f828403f800 nid=0x554d > runnable > "G1 Concurrent Refinement Thread#4" prio=10 tid=0x00007f828403e000 nid=0x554c > runnable > "VM Periodic Task Thread" prio=10 tid=0x00007f82842b8800 nid=0x555b waiting > on condition > JNI global references: 358 > {code} > Jstack of driver: > {code} > Full thread dump OpenJDK 64-Bit Server VM (24.111-b01 mixed mode): > "DestroyJavaVM" prio=10 tid=0x00007f0878012000 nid=0x342b waiting on > condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "java-sdk-progress-listener-callback-thread" daemon prio=10 > tid=0x00007f0844007000 nid=0x5da1 waiting on condition [0x00007f07dee9f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c813a350> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "task-result-getter-3" daemon prio=10 tid=0x00007f082c017800 nid=0x3715 > waiting on condition [0x00007f07df4a5000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0275eb8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "task-result-getter-2" daemon prio=10 tid=0x00007f083400e000 nid=0x370c > waiting on condition [0x00007f07deb9c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0275eb8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "task-result-getter-1" daemon prio=10 tid=0x00007f0830014800 nid=0x3709 > waiting on condition [0x00007f07df3a4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0275eb8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "task-result-getter-0" daemon prio=10 tid=0x00007f083400d000 nid=0x36c2 > waiting on condition [0x00007f07dea9b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0275eb8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-3" daemon prio=10 tid=0x00007f0844005800 nid=0x34a6 runnable > [0x00007f07e49af000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c027f6b0> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000000c027f710> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c027f608> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-2" daemon prio=10 tid=0x00007f0844004800 nid=0x34a3 runnable > [0x00007f07e84c6000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c0283cb8> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000000c0284d28> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c0283c20> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-1" daemon prio=10 tid=0x00007f0844002000 nid=0x34a0 runnable > [0x00007f07d8f01000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c0285398> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000000c02853f8> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c0285300> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-3" daemon prio=10 tid=0x00007f0820004800 nid=0x349e runnable > [0x00007f07db004000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c02e3d48> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000000c02e4db8> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c02e3cb0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-2" daemon prio=10 tid=0x00007f0820004000 nid=0x349c runnable > [0x00007f07df5a6000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c02e5428> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000000c02e5488> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c02e5390> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-1" daemon prio=10 tid=0x00007f0820003000 nid=0x349b runnable > [0x00007f07df6a7000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c02f0af0> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000000c02f1b60> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c02f0a58> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "java-sdk-http-connection-reaper" daemon prio=10 tid=0x00007f0879fcb000 > nid=0x348c waiting on condition [0x00007f07dd314000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:112) > "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" > daemon prio=10 tid=0x00007f0878ad7800 nid=0x348b in Object.wait() > [0x00007f07de697000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000000c6b4da98> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at > org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063) > at java.lang.Thread.run(Thread.java:745) > "ForkJoinPool-1-worker-21" daemon prio=10 tid=0x00007f0808002000 nid=0x347e > waiting on condition [0x00007f07df1a2000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c6d4db68> (a > scala.concurrent.forkjoin.ForkJoinPool) > at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > "threadDeathWatcher-2-1" daemon prio=10 tid=0x00007f0860024800 nid=0x3470 > waiting on condition [0x00007f07e7eb7000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at > io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137) > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > "pool-19-thread-1" daemon prio=10 tid=0x00007f0879d9f800 nid=0x346a waiting > on condition [0x00007f07e85c7000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c2825fc8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "SparkListenerBus" daemon prio=10 tid=0x00007f0879a4b800 nid=0x3468 waiting > on condition [0x00007f07e86c8000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0119688> (a > java.util.concurrent.Semaphore$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) > at > org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:80) > at > org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79) > at > org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) > at > org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78) > at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1249) > at > org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77) > "context-cleaner-periodic-gc" daemon prio=10 tid=0x00007f0879a45800 > nid=0x3467 waiting on condition [0x00007f07e87c9000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0544fe8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Spark Context Cleaner" daemon prio=10 tid=0x00007f0879a44800 nid=0x3466 in > Object.wait() [0x00007f07e88ca000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000000c0544b88> (a java.lang.ref.ReferenceQueue$Lock) > at > org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:175) > at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1249) > at > org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:172) > at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:67) > "Scheduler-214815552" prio=10 tid=0x00007f0804018000 nid=0x3465 waiting on > condition [0x00007f07e89cb000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c031b528> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-0" daemon prio=10 tid=0x00007f087996d800 nid=0x3464 runnable > [0x00007f07e8acc000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c0290de0> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000000c029f290> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c0290d48> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "MesosCoarseGrainedSchedulerBackend-mesos-driver" daemon prio=10 > tid=0x00007f08798a4000 nid=0x3463 runnable [0x00007f07e8dd3000] > java.lang.Thread.State: RUNNABLE > at org.apache.mesos.MesosSchedulerDriver.join(Native Method) > at > org.apache.mesos.MesosSchedulerDriver.run(MesosSchedulerDriver.java:168) > at > org.apache.spark.scheduler.cluster.mesos.MesosSchedulerUtils$$anon$1.run(MesosSchedulerUtils.scala:120) > "driver-revive-thread" daemon prio=10 tid=0x00007f0830003000 nid=0x3459 > waiting on condition [0x00007f07ed6dd000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c02775c8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "dag-scheduler-event-loop" daemon prio=10 tid=0x00007f0879873800 nid=0x3458 > waiting on condition [0x00007f07ed7de000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c026e200> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:489) > at > java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:678) > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46) > "Timer-0" daemon prio=10 tid=0x00007f0879821000 nid=0x3457 in Object.wait() > [0x00007f083c846000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000000c0267430> (a java.util.TaskQueue) > at java.lang.Object.wait(Object.java:503) > at java.util.TimerThread.mainLoop(Timer.java:526) > - locked <0x00000000c0267430> (a java.util.TaskQueue) > at java.util.TimerThread.run(Timer.java:505) > "netty-rpc-env-timeout" daemon prio=10 tid=0x00007f080400e800 nid=0x3456 > waiting on condition [0x00007f083c645000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c02e1338> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "heartbeat-receiver-event-loop-thread" daemon prio=10 tid=0x00007f0834005800 > nid=0x3455 waiting on condition [0x00007f083e277000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c02764f8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "SparkUI-42" daemon prio=10 tid=0x00007f08797ba000 nid=0x3453 waiting on > condition [0x00007f083da2f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0312500> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) > at java.lang.Thread.run(Thread.java:745) > "SparkUI-41" daemon prio=10 tid=0x00007f08797b8000 nid=0x3452 waiting on > condition [0x00007f083db30000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0312500> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) > at java.lang.Thread.run(Thread.java:745) > "SparkUI-40" daemon prio=10 tid=0x00007f08797b6000 nid=0x3451 waiting on > condition [0x00007f083dc31000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0312500> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) > at java.lang.Thread.run(Thread.java:745) > "SparkUI-39" daemon prio=10 tid=0x00007f08797b3800 nid=0x3450 waiting on > condition [0x00007f083dd32000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0312500> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) > at java.lang.Thread.run(Thread.java:745) > "SparkUI-38-acceptor-0@6100756c-ServerConnector@651d3734{HTTP/1.1}{0.0.0.0:4040}" > daemon prio=10 tid=0x00007f08797b1800 nid=0x344f runnable > [0x00007f083de33000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) > - locked <0x00000000c0312cc8> (a java.lang.Object) > at > org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:377) > at > org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Thread.java:745) > "SparkUI-37" daemon prio=10 tid=0x00007f08797af800 nid=0x344e waiting on > condition [0x00007f083df34000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0312500> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) > at java.lang.Thread.run(Thread.java:745) > "SparkUI-36-selector-ServerConnectorManager@13aba9a3/1" daemon prio=10 > tid=0x00007f08797ae000 nid=0x344d runnable [0x00007f083e035000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c031b668> (a sun.nio.ch.Util$2) > - locked <0x00000000c031b658> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c03130a0> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) > at > org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) > at > org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) > at > org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Thread.java:745) > "SparkUI-35-selector-ServerConnectorManager@13aba9a3/0" daemon prio=10 > tid=0x00007f08797ac800 nid=0x344c runnable [0x00007f083e136000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c0315cd0> (a sun.nio.ch.Util$2) > - locked <0x00000000c0315cc0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c0315b38> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) > at > org.spark_project.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:601) > at > org.spark_project.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:550) > at > org.spark_project.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Thread.java:745) > "map-output-dispatcher-7" daemon prio=10 tid=0x00007f0879543000 nid=0x344a > waiting on condition [0x00007f083e378000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0268050> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "map-output-dispatcher-6" daemon prio=10 tid=0x00007f0879540800 nid=0x3449 > waiting on condition [0x00007f083e479000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0268050> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "map-output-dispatcher-5" daemon prio=10 tid=0x00007f087953e800 nid=0x3448 > waiting on condition [0x00007f083e57a000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0268050> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "map-output-dispatcher-4" daemon prio=10 tid=0x00007f087953c800 nid=0x3447 > waiting on condition [0x00007f083e67b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0268050> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "map-output-dispatcher-3" daemon prio=10 tid=0x00007f087953b000 nid=0x3446 > waiting on condition [0x00007f083e77c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0268050> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "map-output-dispatcher-2" daemon prio=10 tid=0x00007f0879539800 nid=0x3445 > waiting on condition [0x00007f083e87d000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0268050> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "map-output-dispatcher-1" daemon prio=10 tid=0x00007f0879532800 nid=0x3444 > waiting on condition [0x00007f083e97e000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0268050> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "map-output-dispatcher-0" daemon prio=10 tid=0x00007f0879531000 nid=0x3443 > waiting on condition [0x00007f083ea7f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c0268050> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "shuffle-server-0" daemon prio=10 tid=0x00007f0879515800 nid=0x3442 runnable > [0x00007f083eb80000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000000c02f24c0> (a > io.netty.channel.nio.SelectedSelectionKeySet) > - locked <0x00000000c02fdbf0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000000c02f2428> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-3" daemon prio=10 tid=0x00007f087937c800 nid=0x3440 > waiting on condition [0x00007f083ec8b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c02c10b8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-2" daemon prio=10 tid=0x00007f087937a800 nid=0x343f > waiting on condition [0x00007f083ed8c000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c02c10b8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-1" daemon prio=10 tid=0x00007f0879379800 nid=0x343e > waiting on condition [0x00007f083ee8d000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c02c10b8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "dispatcher-event-loop-0" daemon prio=10 tid=0x00007f0879378800 nid=0x343d > waiting on condition [0x00007f083f190000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000000c02c10b8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:207) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" daemon prio=10 tid=0x00007f08780b6000 nid=0x3436 runnable > [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=10 tid=0x00007f08780b4000 nid=0x3435 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=10 tid=0x00007f08780b1000 nid=0x3434 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=10 tid=0x00007f08780ae800 nid=0x3433 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=10 tid=0x00007f0878084000 nid=0x3432 in Object.wait() > [0x00007f0874af9000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000000c0015cc8> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=10 tid=0x00007f0878082000 nid=0x3431 in > Object.wait() [0x00007f0874bfa000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000000c0015c98> (a java.lang.ref.Reference$Lock) > "VM Thread" prio=10 tid=0x00007f087807d800 nid=0x3430 runnable > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f0878028000 nid=0x342c > runnable > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f087802a000 nid=0x342d > runnable > "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f087802c000 nid=0x342e > runnable > "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f087802d800 nid=0x342f > runnable > "VM Periodic Task Thread" prio=10 tid=0x00007f08780c1000 nid=0x3437 waiting > on condition > JNI global references: 309 > Heap > PSYoungGen total 136192K, used 39381K [0x00000000eaa80000, > 0x00000000f6b00000, 0x0000000100000000) > eden space 135680K, 29% used > [0x00000000eaa80000,0x00000000ed0f56b0,0x00000000f2f00000) > from space 512K, 0% used > [0x00000000f2f00000,0x00000000f2f00000,0x00000000f2f80000) > to space 30720K, 0% used > [0x00000000f4d00000,0x00000000f4d00000,0x00000000f6b00000) > ParOldGen total 243200K, used 133233K [0x00000000c0000000, > 0x00000000ced80000, 0x00000000eaa80000) > object space 243200K, 54% used > [0x00000000c0000000,0x00000000c821c438,0x00000000ced80000) > PSPermGen total 119296K, used 74568K [0x00000000b0000000, > 0x00000000b7480000, 0x00000000c0000000) > object space 119296K, 62% used > [0x00000000b0000000,0x00000000b48d2208,0x00000000b7480000) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org