So this is pretty weird, and my debugging hasn't made much progress, so I
thought I'd ask for help.
I have a medium size dataset, but I am developing my code against a much
smaller version since it is much smaller and faster to work with. I have
deployed a cluster on EC2 using the default scripts and the AMI and loaded
my simple dataset into the ephermeral HDFS.
When I ssh into the master node and run my application using "local",
everything works. However, if I just change to using distributed spark
(spark://...:7077) then everything starts up, but the shuffle operations
start failing with the error:
3/12/02 06:47:33 INFO spark.MapOutputTrackerActor: Asked to send map output
locations for shuffle 0 to ip-10-38-11-59.ec2.internal:39992
...
13/12/02 06:49:19 WARN storage.BlockManagerMasterActor: Removing
BlockManager BlockManagerId(0, ip-10-38-11-59.ec2.internal, 46556, 0) with
no recent heart beats: 51656ms exceeds 45000ms
I looked around the prior documentation and put the extra options into
SPARK_JAVA_OPTS, and increased them to 90000 from the 30000 in the example
but I get the same error.
-Dspark.worker.timeout=90000 -Dspark.akka.timeout=90000
-Dspark.storage.blockManagerHeartBeatMs=90000
-Dspark.akka.retry.wait=90000 -Dspark.akka.frameSize=30000
-Dsun.rmi.dgc.server.gcInterval=3600000
I don't think that this is a memory issue because the dataset fits on the
master alone and I am successfully able to run my program with "local". I
checked the stdout and stderr on the worker, and one of the few times that
there was a stacktrace, it was from here:
148 private def askDriverWithReply[T](message: Any): T = {
149 // TODO: Consider removing multiple attempts
150 if (driverActor == null) {
151 throw new SparkException("Error sending message to BlockManager
as driverActor is null " +
152 "[message = " + message + "]")
153 }
154 var attempts = 0
155 var lastException: Exception = null
156 while (attempts < AKKA_RETRY_ATTEMPTS) {
157 attempts += 1
158 try {
159 val future = driverActor.ask(message)(timeout)
160 val result = Await.result(future, timeout)
161 if (result == null) {
162 throw new SparkException("BlockManagerMaster returned null")
163 }
164 return result.asInstanceOf[T]
165 } catch {
166 case ie: InterruptedException => throw ie
167 case e: Exception =>
168 lastException = e
169 logWarning("Error sending message to BlockManagerMaster in "
+ attempts + "attempts", e)
170 }
171 Thread.sleep(AKKA_RETRY_INTERVAL_MS)
172 }
173
174 throw new SparkException(
175 "Error sending message to BlockManagerMaster [message = " +
message + "]", lastException)
176 }
Most of the time, though, the worker just hangs. I can't run anything else
against that master because there are no resources available. When I try to
stop the workers using stop-slaves.sh, they don't stop. The only way to
recover the cluster is to use "kill -9 <pid>" on the worker processes,
which does work.
So I took a quick look at one of the hung worker processes using jstack.
The output is attached. As you can see, I have tried to shut down the
process multiple times, and the SIGTERM handlers are stuck waiting. I think
that this may be the reason that the workers are not responsive for block
handling as well.
If I read this correctly, the handler is waiting on 0x00000007e025d468
which is locked by spark.deploy.worker.ExecutorRunner$$anon$2. And that is
waiting for some unixprocess?
Thanks,
Shankari
2013-12-0 23:55:46
Full thread dump OpenJDK 64-Bit Server VM (24.45-b08 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007f5980009000 nid=0x405d waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"SIGTERM handler" daemon prio=10 tid=0x00007f5980001800 nid=0x4049 waiting for
monitor entry [0x00007f5998a62000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:212)
- waiting to lock <0x00000007e02720f0> (a java.lang.Class for
java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:744)
"SIGTERM handler" daemon prio=10 tid=0x00007f5980003000 nid=0x402d waiting for
monitor entry [0x00007f599865e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:212)
- waiting to lock <0x00000007e02720f0> (a java.lang.Class for
java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:744)
"sparkWorker-akka.actor.default-dispatcher-10" daemon prio=10
tid=0x00007f596802e000 nid=0x3d0c waiting on condition [0x00007f599a24d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e0286028> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at akka.jsr166y.ForkJoinPool.scan(ForkJoinPool.java:1594)
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
"SIGTERM handler" daemon prio=10 tid=0x00007f5980007000 nid=0x3cea waiting for
monitor entry [0x00007f5998860000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:212)
- waiting to lock <0x00000007e02720f0> (a java.lang.Class for
java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:744)
"SIGTERM handler" daemon prio=10 tid=0x00007f5980005000 nid=0x3cce waiting for
monitor entry [0x00007f5998961000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:212)
- waiting to lock <0x00000007e02720f0> (a java.lang.Class for
java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:744)
"Thread-9" daemon prio=10 tid=0x00007f596c063000 nid=0x3cb0 in Object.wait()
[0x00007f599b35e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007e025e150> (a java.lang.UNIXProcess)
at java.lang.Object.wait(Object.java:503)
at java.lang.UNIXProcess.waitFor(UNIXProcess.java:210)
- locked <0x00000007e025e150> (a java.lang.UNIXProcess)
at
org.apache.spark.deploy.worker.ExecutorRunner$$anon$2.run(ExecutorRunner.scala:69)
"SIGTERM handler" daemon prio=10 tid=0x00007f5980002800 nid=0x3caf in
Object.wait() [0x00007f599a550000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007e025d468> (a
org.apache.spark.deploy.worker.ExecutorRunner$$anon$2)
at java.lang.Thread.join(Thread.java:1280)
- locked <0x00000007e025d468> (a
org.apache.spark.deploy.worker.ExecutorRunner$$anon$2)
at java.lang.Thread.join(Thread.java:1354)
at
java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
at
java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000007e02720f0> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:744)
"sparkWorker-akka.actor.default-dispatcher-9" daemon prio=10
tid=0x00007f5964002000 nid=0x3739 waiting on condition [0x00007f599a14c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e0286028> (a
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at akka.jsr166y.ForkJoinPool.scan(ForkJoinPool.java:1594)
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
"sparkWorker-akka.actor.default-dispatcher-6" daemon prio=10
tid=0x00007f596803a800 nid=0x30e5 in Object.wait() [0x00007f599a650000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007e025e150> (a java.lang.UNIXProcess)
at java.lang.Object.wait(Object.java:503)
at java.lang.UNIXProcess.waitFor(UNIXProcess.java:210)
- locked <0x00000007e025e150> (a java.lang.UNIXProcess)
at
org.apache.spark.deploy.worker.ExecutorRunner.kill(ExecutorRunner.scala:84)
at
org.apache.spark.deploy.worker.Worker$$anonfun$receive$1.apply(Worker.scala:161)
at
org.apache.spark.deploy.worker.Worker$$anonfun$receive$1.apply(Worker.scala:120)
at akka.actor.Actor$class.apply(Actor.scala:318)
at org.apache.spark.deploy.worker.Worker.apply(Worker.scala:39)
at akka.actor.ActorCell.invoke(ActorCell.scala:626)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
at akka.dispatch.Mailbox.run(Mailbox.scala:179)
at
akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
"redirect output to /root/spark/work/app-20131202064720-0000/0/stderr" daemon
prio=10 tid=0x00007f596c062800 nid=0x30a1 runnable [0x00007f5998b63000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00000007e0272058> (a
java.lang.UNIXProcess$ProcessPipeInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.spark.util.Utils$.copyStream(Utils.scala:211)
at
org.apache.spark.deploy.worker.ExecutorRunner$$anon$3.run(ExecutorRunner.scala:135)
"redirect output to /root/spark/work/app-20131202064720-0000/0/stdout" daemon
prio=10 tid=0x00007f596c057000 nid=0x309f runnable [0x00007f599b25d000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00000007e0272030> (a
java.lang.UNIXProcess$ProcessPipeInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.spark.util.Utils$.copyStream(Utils.scala:211)
at
org.apache.spark.deploy.worker.ExecutorRunner$$anon$3.run(ExecutorRunner.scala:135)
"process reaper" daemon prio=10 tid=0x00007f596c04a000 nid=0x3093 runnable
[0x00007f59bc040000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.waitForProcessExit(Native Method)
at java.lang.UNIXProcess.access$200(UNIXProcess.java:54)
at java.lang.UNIXProcess$3.run(UNIXProcess.java:174)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"Hashed wheel timer #1" daemon prio=10 tid=0x00007f5960118000 nid=0x308a
waiting on condition [0x00007f599a752000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:455)
at
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:372)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at java.lang.Thread.run(Thread.java:744)
"qtp436478131-22" daemon prio=10 tid=0x00007f596010d000 nid=0x3089 waiting on
condition [0x00007f599a853000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e02e26b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
"qtp436478131-21" daemon prio=10 tid=0x00007f596010a800 nid=0x3088 waiting on
condition [0x00007f599a954000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e02e26b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
"qtp436478131-20" daemon prio=10 tid=0x00007f5960108800 nid=0x3087 waiting on
condition [0x00007f599aa55000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e02e26b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
"qtp436478131-19" daemon prio=10 tid=0x00007f5960107000 nid=0x3086 waiting on
condition [0x00007f599ab56000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e02e26b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
"qtp436478131-18" daemon prio=10 tid=0x00007f5960106000 nid=0x3085 waiting on
condition [0x00007f599ac57000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e02e26b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
"qtp436478131-17" daemon prio=10 tid=0x00007f5960101800 nid=0x3084 waiting on
condition [0x00007f599ad58000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e02e26b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:744)
"qtp436478131-16 Acceptor0 [email protected]:8081" daemon prio=10
tid=0x00007f59600ff800 nid=0x3083 runnable [0x00007f599ae59000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked <0x00000007e02e1208> (a java.lang.Object)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
"qtp436478131-15 Selector0" daemon prio=10 tid=0x00007f59600d4800 nid=0x3082
runnable [0x00007f599af5a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007e02e0d98> (a sun.nio.ch.Util$2)
- locked <0x00000007e02e0d88> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007e02e0b60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
"sparkWorker-2" daemon prio=10 tid=0x00007f59b83f5800 nid=0x3081 runnable
[0x00007f599b05b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007e0198138> (a sun.nio.ch.Util$2)
- locked <0x00000007e0198128> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007e0197d30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"sparkWorker-scheduler-2" daemon prio=10 tid=0x00007f59b8311000 nid=0x3080
waiting on condition [0x00007f599b15c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.jboss.netty.akka.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:418)
at
org.jboss.netty.akka.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:318)
at java.lang.Thread.run(Thread.java:744)
"Service Thread" daemon prio=10 tid=0x00007f59b80a1000 nid=0x307b runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f59b809e800 nid=0x307a waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f59b809b800 nid=0x3079 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f59b8099800 nid=0x3078 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f59b8077000 nid=0x3077 in Object.wait()
[0x00007f599b979000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007e0008db0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000007e0008db0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
"Reference Handler" daemon prio=10 tid=0x00007f59b8075000 nid=0x3076 in
Object.wait() [0x00007f599ba7a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007e0008770> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000007e0008770> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007f59b8009800 nid=0x3072 waiting on condition
[0x00007f59c0db4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007e0309760> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at
akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:736)
at
akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:705)
at akka.dispatch.Await$.ready(Future.scala:64)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:604)
at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:605)
at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:200)
at org.apache.spark.deploy.worker.Worker.main(Worker.scala)
"VM Thread" prio=10 tid=0x00007f59b8070800 nid=0x3075 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f59b801f800 nid=0x3073
runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f59b8021000 nid=0x3074
runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f59b80ab800 nid=0x307c waiting on
condition
JNI global references: 252
