I think your thread dump for the master is actually just a thread dump for
SBT that is waiting on a forked driver program.
...
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007fed624ff528> (a java.lang.UNIXProcess)
at java.lang.Object.wait(Object.java:503)
at java.lang.UNIXProcess.waitFor(UNIXProcess.java:263)
- locked <0x00007fed624ff528> (a java.lang.UNIXProcess)
at sbt.SimpleProcess.exitValue(ProcessImpl.scala:377)
...
Additionally, no work seems to be happening on the worker. I think you
want to jstack the process "ForkMain".
On Wed, Apr 8, 2015 at 2:37 AM, Hao Ren <inv...@gmail.com> wrote:
> Hi Michael，
>
> In fact, I find that all workers are hanging when SQL/DF join is running.
>
> So I picked the master and one of the workers. jstack is the following:
>
> Master
>
> ------------------------------------------------------------------------------------------------------------------------
> 2015-04-08 09:08:22
> Full thread dump OpenJDK 64-Bit Server VM (24.65-b04 mixed mode):
>
> "Thread-3" prio=10 tid=0x00007fdbe0013000 nid=0x2836 runnable
> [0x00007fdd0cd89000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:272)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00007fed62503a00> (a
> java.lang.UNIXProcess$ProcessPipeInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> - locked <0x00007fed95527048> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> - locked <0x00007fed95527048> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> sbt.BasicIO$$anonfun$processFully$1$$anonfun$apply$8.apply(ProcessImpl.scala:58)
> at
> sbt.BasicIO$$anonfun$processFully$1$$anonfun$apply$8.apply(ProcessImpl.scala:58)
> at sbt.BasicIO$.readFully$1(ProcessImpl.scala:63)
> at sbt.BasicIO$.processLinesFully(ProcessImpl.scala:69)
> at sbt.BasicIO$$anonfun$processFully$1.apply(ProcessImpl.scala:58)
> at sbt.BasicIO$$anonfun$processFully$1.apply(ProcessImpl.scala:55)
> at sbt.SimpleProcessBuilder$$anonfun$4.apply$mcV$sp(ProcessImpl.scala:357)
> at sbt.Spawn$$anon$3.run(ProcessImpl.scala:17)
>
> "Thread-2" prio=10 tid=0x00007fdbe0010000 nid=0x2835 runnable
> [0x00007fdd0d995000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:272)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00007fed625018b8> (a
> java.lang.UNIXProcess$ProcessPipeInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
> - locked <0x00007fed8cc9d8b8> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:154)
> at java.io.BufferedReader.readLine(BufferedReader.java:317)
> - locked <0x00007fed8cc9d8b8> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at
> sbt.BasicIO$$anonfun$processFully$1$$anonfun$apply$8.apply(ProcessImpl.scala:58)
> at
> sbt.BasicIO$$anonfun$processFully$1$$anonfun$apply$8.apply(ProcessImpl.scala:58)
> at sbt.BasicIO$.readFully$1(ProcessImpl.scala:63)
> at sbt.BasicIO$.processLinesFully(ProcessImpl.scala:69)
> at sbt.BasicIO$$anonfun$processFully$1.apply(ProcessImpl.scala:58)
> at sbt.BasicIO$$anonfun$processFully$1.apply(ProcessImpl.scala:55)
> at sbt.SimpleProcessBuilder$$anonfun$3.apply$mcV$sp(ProcessImpl.scala:354)
> at sbt.Spawn$$anon$3.run(ProcessImpl.scala:17)
>
> "pool-5-thread-5" prio=10 tid=0x00007ff16034a000 nid=0x2832 waiting on
> condition [0x00007fdd0da96000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feacd0621d0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "pool-5-thread-4" prio=10 tid=0x00007ff160349800 nid=0x2831 waiting on
> condition [0x00007fdd0d894000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feacd0621d0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "pool-5-thread-3" prio=10 tid=0x00007ff160349000 nid=0x2830 waiting on
> condition [0x00007fdd0d692000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feacd0621d0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "pool-5-thread-2" prio=10 tid=0x00007ff160348000 nid=0x282f in
> Object.wait() [0x00007fdd0d38e000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007fed624ff528> (a java.lang.UNIXProcess)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.UNIXProcess.waitFor(UNIXProcess.java:263)
> - locked <0x00007fed624ff528> (a java.lang.UNIXProcess)
> at sbt.SimpleProcess.exitValue(ProcessImpl.scala:377)
> at sbt.ForkRun.run(Run.scala:32)
> at
> sbt.Defaults$$anonfun$runMainTask$1$$anonfun$apply$34$$anonfun$apply$35.apply(Defaults.scala:697)
> at
> sbt.Defaults$$anonfun$runMainTask$1$$anonfun$apply$34$$anonfun$apply$35.apply(Defaults.scala:695)
> at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
> at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
> at sbt.std.Transform$$anon$4.work(System.scala:63)
> at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
> at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
> at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
> at sbt.Execute.work(Execute.scala:235)
> at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
> at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
> at
> sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
> at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "pool-5-thread-1" prio=10 tid=0x00007ff160347800 nid=0x282e waiting on
> condition [0x00007fdd0d08c000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feacd0621d0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "Attach Listener" daemon prio=10 tid=0x00007fdc28001000 nid=0x282d waiting
> on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "ForkJoinPool-1-worker-15" daemon prio=10 tid=0x00007fdbdc004000
> nid=0x2812 waiting on condition [0x00007fdcff4ad000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feb0cd24a08> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "ForkJoinPool-1-worker-1" daemon prio=10 tid=0x00007fdbf409f800 nid=0x2811
> waiting on condition [0x00007fdcff5ae000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feb0cd24a08> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "ForkJoinPool-1-worker-5" daemon prio=10 tid=0x00007fdbe4026800 nid=0x2810
> waiting on condition [0x00007fdd0c150000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feb0cd24a08> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "ForkJoinPool-1-worker-19" daemon prio=10 tid=0x00007fdbd0008800
> nid=0x280e waiting on condition [0x00007fdd0c352000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feb0cd24a08> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "ForkJoinPool-1-worker-23" daemon prio=10 tid=0x00007fdbf8004800
> nid=0x280d waiting on condition [0x00007fdd0c453000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feb0cd24a08> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "ForkJoinPool-1-worker-31" daemon prio=10 tid=0x00007fdbd0006000
> nid=0x280c waiting on condition [0x00007fdd0cf8b000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feb0cd24a08> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "ForkJoinPool-1-worker-13" daemon prio=10 tid=0x00007fdbf8004000
> nid=0x280b waiting on condition [0x00007fdd0ce8a000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feb0cd24a08> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "ForkJoinPool-1-worker-3" daemon prio=10 tid=0x00007fdbe4024800 nid=0x2808
> waiting on condition [0x00007fdd0d490000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feb0cd24a08> (a
> scala.concurrent.forkjoin.ForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "process reaper" daemon prio=10 tid=0x00007ff160886000 nid=0x27af runnable
> [0x00007fdd0dadd000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.UNIXProcess.waitForProcessExit(Native Method)
> at java.lang.UNIXProcess.access$500(UNIXProcess.java:54)
> at java.lang.UNIXProcess$4.run(UNIXProcess.java:227)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" daemon prio=10 tid=0x00007ff1602a9800 nid=0x27ac runnable
> [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=10 tid=0x00007ff1602a7000 nid=0x27ab
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=10 tid=0x00007ff1602a4000 nid=0x27aa
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007ff1602a2000 nid=0x27a9
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007ff160275000 nid=0x27a8 in
> Object.wait() [0x00007fdd0e6e5000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007fefb26280a8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00007fefb26280a8> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" daemon prio=10 tid=0x00007ff160273000 nid=0x27a7 in
> Object.wait() [0x00007fdd0e7e6000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007fefb263c0b8> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> - locked <0x00007fefb263c0b8> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x00007ff160009000 nid=0x2798 waiting on condition
> [0x00007ff1695b3000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00007feacd062538> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
> at
> sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:188)
> at sbt.Execute.next$1(Execute.scala:83)
> at sbt.Execute.processAll(Execute.scala:86)
> at sbt.Execute.runKeep(Execute.scala:66)
> at sbt.EvaluateTask$.liftedTree1$1(EvaluateTask.scala:342)
> at sbt.EvaluateTask$.run$1(EvaluateTask.scala:341)
> at sbt.EvaluateTask$.runTask(EvaluateTask.scala:361)
> at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:64)
> at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:62)
> at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:293)
> at sbt.Aggregation$.timedRun(Aggregation.scala:62)
> at sbt.Aggregation$.runTasks(Aggregation.scala:71)
> at
> sbt.Aggregation$$anonfun$applyDynamicTasks$1.apply(Aggregation.scala:112)
> at
> sbt.Aggregation$$anonfun$applyDynamicTasks$1.apply(Aggregation.scala:110)
> at
> sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
> at
> sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:60)
> at
> sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:244)
> at
> sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:241)
> at sbt.Command$.process(Command.scala:92)
> at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
> at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:98)
> at sbt.State$$anon$1.process(State.scala:184)
> at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
> at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:98)
> at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
> at sbt.MainLoop$.next(MainLoop.scala:98)
> at sbt.MainLoop$.run(MainLoop.scala:91)
> at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:70)
> at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:65)
> at sbt.Using.apply(Using.scala:24)
> at sbt.MainLoop$.runWithNewLog(MainLoop.scala:65)
> at sbt.MainLoop$.runAndClearLast(MainLoop.scala:48)
> at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:32)
> at sbt.MainLoop$.runLogged(MainLoop.scala:24)
> at sbt.StandardMain$.runManaged(Main.scala:53)
> at sbt.xMain.run(Main.scala:28)
> at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
> at xsbt.boot.Launch$.withContextLoader(Launch.scala:128)
> at xsbt.boot.Launch$.run(Launch.scala:109)
> at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
> at xsbt.boot.Launch$.launch(Launch.scala:117)
> at xsbt.boot.Launch$.apply(Launch.scala:18)
> at xsbt.boot.Boot$.runImpl(Boot.scala:41)
> at xsbt.boot.Boot$.main(Boot.scala:17)
> at xsbt.boot.Boot.main(Boot.scala)
>
> "VM Thread" prio=10 tid=0x00007ff16026f000 nid=0x27a6 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007ff16001e800 nid=0x2799
> runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007ff160020800 nid=0x279a
> runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007ff160022800 nid=0x279b
> runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007ff160024000 nid=0x279c
> runnable
>
> "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007ff160026000 nid=0x279d
> runnable
>
> "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007ff160028000 nid=0x279e
> runnable
>
> "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007ff16002a000 nid=0x279f
> runnable
>
> "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007ff16002b800 nid=0x27a0
> runnable
>
> "GC task thread#8 (ParallelGC)" prio=10 tid=0x00007ff16002d800 nid=0x27a1
> runnable
>
> "GC task thread#9 (ParallelGC)" prio=10 tid=0x00007ff16002f800 nid=0x27a2
> runnable
>
> "GC task thread#10 (ParallelGC)" prio=10 tid=0x00007ff160031000 nid=0x27a3
> runnable
>
> "GC task thread#11 (ParallelGC)" prio=10 tid=0x00007ff160033000 nid=0x27a4
> runnable
>
> "GC task thread#12 (ParallelGC)" prio=10 tid=0x00007ff160035000 nid=0x27a5
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007ff1602b4000 nid=0x27ad
> waiting on condition
>
> JNI global references: 187
>
>
> Slave
>
> ------------------------------------------------------------------------------------------------------------------------------------------------
> 2015-04-08 09:10:02
> Full thread dump OpenJDK 64-Bit Server VM (24.65-b04 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x00007f1454001000 nid=0x5640 waiting
> on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "File appending thread for
> /root/spark/work/app-20150408090735-0010/2/stderr" daemon prio=10
> tid=0x00007f1410003000 nid=0x546d runnable [0x00007f14593d0000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:272)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00000000f844db20> (a
> java.lang.UNIXProcess$ProcessPipeInputStream)
> at java.io.FilterInputStream.read(FilterInputStream.java:107)
> at
> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
> at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
> at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
> at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618)
> at
> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
>
> "File appending thread for
> /root/spark/work/app-20150408090735-0010/2/stdout" daemon prio=10
> tid=0x00007f1410002000 nid=0x546c runnable [0x00007f14594d1000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:272)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x00000000f844ba58> (a
> java.lang.UNIXProcess$ProcessPipeInputStream)
> at java.io.FilterInputStream.read(FilterInputStream.java:107)
> at
> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
> at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
> at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
> at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618)
> at
> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
>
> "process reaper" daemon prio=10 tid=0x00007f1410001000 nid=0x544a runnable
> [0x00007f14900cc000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.UNIXProcess.waitForProcessExit(Native Method)
> at java.lang.UNIXProcess.access$500(UNIXProcess.java:54)
> at java.lang.UNIXProcess$4.run(UNIXProcess.java:227)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "ExecutorRunner for app-20150408090735-0010/2" daemon prio=10
> tid=0x00007f143400a800 nid=0x5448 in Object.wait() [0x00007f14592cf000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000f84497a0> (a java.lang.UNIXProcess)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.UNIXProcess.waitFor(UNIXProcess.java:263)
> - locked <0x00000000f84497a0> (a java.lang.UNIXProcess)
> at
> org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:163)
> at
> org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:68)
>
> "Hashed wheel timer #1" daemon prio=10 tid=0x00007f1420001000 nid=0xe93
> waiting on condition [0x00007f14595d2000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
> at
> org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1611672401-32" daemon prio=10 tid=0x00007f14380c5000 nid=0xe92 waiting
> on condition [0x00007f14596d3000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd6104d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1611672401-31" daemon prio=10 tid=0x00007f14380c3000 nid=0xe91 waiting
> on condition [0x00007f14597d4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd6104d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1611672401-30" daemon prio=10 tid=0x00007f14380c1000 nid=0xe90 waiting
> on condition [0x00007f14598d5000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd6104d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1611672401-29" daemon prio=10 tid=0x00007f14380bf800 nid=0xe8f waiting
> on condition [0x00007f14599d6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd6104d8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1611672401-28 Acceptor1 SelectChannelConnector@0.0.0.0:8081" daemon
> prio=10 tid=0x00007f14380be000 nid=0xe8e waiting for monitor entry
> [0x00007f1459ad7000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
> - waiting to lock <0x00000000fd600598> (a java.lang.Object)
> at
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1611672401-27 Acceptor0 SelectChannelConnector@0.0.0.0:8081" daemon
> prio=10 tid=0x00007f14380bc800 nid=0xe8d runnable [0x00007f146c165000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
> - locked <0x00000000fd600598> (a java.lang.Object)
> at
> org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
> at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1611672401-26 Selector1" daemon prio=10 tid=0x00007f14380ba800
> nid=0xe8c runnable [0x00007f146c266000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000fd648118> (a sun.nio.ch.Util$2)
> - locked <0x00000000fd648128> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fd6480d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "qtp1611672401-25 Selector0" daemon prio=10 tid=0x00007f14380b4000
> nid=0xe8b runnable [0x00007f146c367000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000fd624230> (a sun.nio.ch.Util$2)
> - locked <0x00000000fd624240> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fd6241e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O server boss #6" daemon prio=10 tid=0x00007f14300c7000 nid=0xe88
> runnable [0x00007f146c468000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000fd6301c8> (a sun.nio.ch.Util$2)
> - locked <0x00000000fd6301d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fd630180> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #5" daemon prio=10 tid=0x00007f14300c0000 nid=0xe87
> runnable [0x00007f146c569000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000fd6106a0> (a sun.nio.ch.Util$2)
> - locked <0x00000000fd6106b0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fd610658> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #4" daemon prio=10 tid=0x00007f1430043000 nid=0xe86
> runnable [0x00007f146c66a000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000fd6006d0> (a sun.nio.ch.Util$2)
> - locked <0x00000000fd6006e0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fd600688> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O boss #3" daemon prio=10 tid=0x00007f14300c2800 nid=0xe85 runnable
> [0x00007f146c76b000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000fd640318> (a sun.nio.ch.Util$2)
> - locked <0x00000000fd640328> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fd6402d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #2" daemon prio=10 tid=0x00007f1430054800 nid=0xe84
> runnable [0x00007f146c86c000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000fd6182c8> (a sun.nio.ch.Util$2)
> - locked <0x00000000fd6182d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fd618280> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "New I/O worker #1" daemon prio=10 tid=0x00007f1430020800 nid=0xe83
> runnable [0x00007f146c96d000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> - locked <0x00000000fd618538> (a sun.nio.ch.Util$2)
> - locked <0x00000000fd618548> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000fd6184f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> "sparkWorker-akka.remote.default-remote-dispatcher-6" daemon prio=10
> tid=0x00007f149453e800 nid=0xe82 waiting on condition [0x00007f146ca6e000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd640730> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkWorker-akka.remote.default-remote-dispatcher-5" daemon prio=10
> tid=0x00007f1438004800 nid=0xe81 waiting on condition [0x00007f146cb6f000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd640730> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkWorker-akka.actor.default-dispatcher-4" daemon prio=10
> tid=0x00007f1438001800 nid=0xe80 waiting on condition [0x00007f146cc70000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd610a58> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkWorker-akka.actor.default-dispatcher-3" daemon prio=10
> tid=0x00007f1494523000 nid=0xe7f waiting on condition [0x00007f146cd71000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd610a58> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkWorker-akka.actor.default-dispatcher-2" daemon prio=10
> tid=0x00007f1494508000 nid=0xe7e waiting on condition [0x00007f146ce72000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd610a58> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> "sparkWorker-scheduler-1" daemon prio=10 tid=0x00007f1494494800 nid=0xe7d
> waiting on condition [0x00007f146cf73000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
> at java.lang.Thread.run(Thread.java:745)
>
> "Service Thread" daemon prio=10 tid=0x00007f14940b8000 nid=0xe7b runnable
> [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f14940b5800 nid=0xe7a
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f14940b2800 nid=0xe79
> waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f14940b0800 nid=0xe78
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f1494083800 nid=0xe77 in
> Object.wait() [0x00007f146ebfa000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000fd63b038> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000000fd63b038> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>
> "Reference Handler" daemon prio=10 tid=0x00007f1494081800 nid=0xe76 in
> Object.wait() [0x00007f14901f0000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000fd63aa98> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> - locked <0x00000000fd63aa98> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x00007f1494009000 nid=0xe6c waiting on condition
> [0x00007f149cc2f000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fd648430> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:817)
> at
> akka.actor.ActorSystemImpl$TerminationCallbacks.ready(ActorSystem.scala:786)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.ready(package.scala:86)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:642)
> at akka.actor.ActorSystemImpl.awaitTermination(ActorSystem.scala:643)
> at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:531)
> at org.apache.spark.deploy.worker.Worker.main(Worker.scala)
>
> "VM Thread" prio=10 tid=0x00007f149407d000 nid=0xe75 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f149401f000 nid=0xe6d
> runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f1494021000 nid=0xe6e
> runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f1494022800 nid=0xe6f
> runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f1494024800 nid=0xe70
> runnable
>
> "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f1494026800 nid=0xe71
> runnable
>
> "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f1494028000 nid=0xe72
> runnable
>
> "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f149402a000 nid=0xe73
> runnable
>
> "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f149402c000 nid=0xe74
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f14940c2800 nid=0xe7c waiting
> on condition
>
> JNI global references: 230
>
>
> Tell me if anything else is needed.
>
> Thank you.
>
> Hao.
>
>
> On Tue, Apr 7, 2015 at 8:00 PM, Michael Armbrust <mich...@databricks.com>
> wrote:
>
>> The joins here are totally different implementations, but it is worrisome
>> that you are seeing the SQL join hanging. Can you provide more information
>> about the hang? jstack of the driver and a worker that is processing a
>> task would be very useful.
>>
>> On Tue, Apr 7, 2015 at 8:33 AM, Hao Ren <inv...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We have 2 hive tables and want to join one with the other.
>>>
>>> Initially, we ran a sql request on HiveContext. But it did not work. It
>>> was
>>> blocked on 30/600 tasks.
>>> Then we tried to load tables into two DataFrames, we have encountered the
>>> same problem.
>>> Finally, it works with RDD.join. What we have done is basically
>>> transforming
>>> 2 tables into 2 pair RDDs, then calling a join operation. It works great
>>> in
>>> about 500 s.
>>>
>>> However, workaround is just a workaround, since we have to transform hive
>>> tables into RDD. This is really annoying.
>>>
>>> Just wondering whether the underlying code of DF/SQL's join operation is
>>> the
>>> same as rdd's, knowing that there is a syntax analysis layer for DF/SQL,
>>> while RDD's join is straightforward on two pair RDDs.
>>>
>>> SQL request:
>>> ----------------------------------------------------------------------
>>> select v1.receipt_id, v1.sku, v1.amount, v1.qty, v2.discount
>>> from table1 as v1 left join table2 as v2
>>> on v1.receipt_id = v2.receipt_id
>>> where v1.sku != ""
>>>
>>> DataFrame:
>>>
>>> -----------------------------------------------------------------------------------------
>>> val rdd1 = ss.hiveContext.table(table1)
>>> val rdd1Filt = rdd1.filter(rdd1.col("sku") !== "")
>>> val rdd2 = ss.hiveContext.table(table2)
>>> val rddJoin = rdd1Filt.join(rdd2, rdd1Filt("receipt_id") ===
>>> rdd2("receipt_id"))
>>> rddJoin.saveAsTable("testJoinTable", SaveMode.Overwrite)
>>>
>>> RDD workaround in this case is a bit cumbersome, for short, we just
>>> created
>>> 2 RDDs, join, and then apply a new schema on the result RDD. This
>>> approach
>>> works, at least all tasks were finished, while the DF/SQL approach don't.
>>>
>>> Any idea ?
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/The-differentce-between-SparkSql-DataFram-join-and-Rdd-join-tp22407.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>>
>
>
> --
> Hao Ren
>
> {Data, Software} Engineer @ ClaraVista
>
> Paris, France
>
