GitHub user attilapiros opened a pull request:
https://github.com/apache/spark/pull/20474
[SPARK-23235][Core] Add executor Threaddump to api
## What changes were proposed in this pull request?
Extending api with the executor thread dump data.
For this new REST URL is introduced:
- GET
http://localhost:4040/api/v1/applications/{applicationId}/executors/{executorId}/threads
<details>
<summary>Example response:</summary>
``` javascript
[ {
"threadId" : 52,
"threadName" : "context-cleaner-periodic-gc",
"threadState" : "TIMED_WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1385411893})",
"holdingLocks" : [ ]
}, {
"threadId" : 48,
"threadName" : "dag-scheduler-event-loop",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)\njava.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)\norg.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:46)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1138053349})",
"holdingLocks" : [ ]
}, {
"threadId" : 17,
"threadName" : "dispatcher-event-loop-0",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1764626380})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@832743930})" ]
}, {
"threadId" : 18,
"threadName" : "dispatcher-event-loop-1",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1764626380})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@834153999})" ]
}, {
"threadId" : 19,
"threadName" : "dispatcher-event-loop-2",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1764626380})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@664836465})" ]
}, {
"threadId" : 20,
"threadName" : "dispatcher-event-loop-3",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1764626380})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1645557354})" ]
}, {
"threadId" : 21,
"threadName" : "dispatcher-event-loop-4",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1764626380})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1188871851})" ]
}, {
"threadId" : 22,
"threadName" : "dispatcher-event-loop-5",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1764626380})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@920926249})" ]
}, {
"threadId" : 23,
"threadName" : "dispatcher-event-loop-6",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1764626380})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@355222677})" ]
}, {
"threadId" : 24,
"threadName" : "dispatcher-event-loop-7",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1764626380})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1589745212})" ]
}, {
"threadId" : 49,
"threadName" : "driver-heartbeater",
"threadState" : "TIMED_WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1602885835})",
"holdingLocks" : [ ]
}, {
"threadId" : 53,
"threadName" : "element-tracking-store-worker",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1439439099})",
"holdingLocks" : [ ]
}, {
"threadId" : 3,
"threadName" : "Finalizer",
"threadState" : "WAITING",
"stackTrace" : "java.lang.Object.wait(Native
Method)\njava.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)\njava.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)\njava.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)",
"blockedByLock" : "Lock(java.lang.ref.ReferenceQueue$Lock@1213098236})",
"holdingLocks" : [ ]
}, {
"threadId" : 15,
"threadName" : "ForkJoinPool-1-worker-13",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\nscala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)\nscala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\nscala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)",
"blockedByLock" :
"Lock(scala.concurrent.forkjoin.ForkJoinPool@380286413})",
"holdingLocks" : [ ]
}, {
"threadId" : 45,
"threadName" : "heartbeat-receiver-event-loop-thread",
"threadState" : "TIMED_WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@715135812})",
"holdingLocks" : [ ]
}, {
"threadId" : 1,
"threadName" : "main",
"threadState" : "RUNNABLE",
"stackTrace" : "java.io.FileInputStream.read0(Native
Method)\njava.io.FileInputStream.read(FileInputStream.java:207)\nscala.tools.jline_embedded.internal.NonBlockingInputStream.read(NonBlockingInputStream.java:169)
=> holding
Monitor(scala.tools.jline_embedded.internal.NonBlockingInputStream@46248392})\nscala.tools.jline_embedded.internal.NonBlockingInputStream.read(NonBlockingInputStream.java:137)\nscala.tools.jline_embedded.internal.NonBlockingInputStream.read(NonBlockingInputStream.java:246)\nscala.tools.jline_embedded.internal.InputStreamReader.read(InputStreamReader.java:261)
=> holding
Monitor(scala.tools.jline_embedded.internal.NonBlockingInputStream@46248392})\nscala.tools.jline_embedded.internal.InputStreamReader.read(InputStreamReader.java:198)
=> holding
Monitor(scala.tools.jline_embedded.internal.NonBlockingInputStream@46248392})\nscala.tools.jline_embedded.console.ConsoleReader.readCharacter(ConsoleReader.java:2145)\nscala.tools.jline_embedded.console.ConsoleReader
.readLine(ConsoleReader.java:2349)\nscala.tools.jline_embedded.console.ConsoleReader.readLine(ConsoleReader.java:2269)\nscala.tools.nsc.interpreter.jline_embedded.InteractiveReader.readOneLine(JLineReader.scala:57)\nscala.tools.nsc.interpreter.InteractiveReader$$anonfun$readLine$2.apply(InteractiveReader.scala:37)\nscala.tools.nsc.interpreter.InteractiveReader$$anonfun$readLine$2.apply(InteractiveReader.scala:37)\nscala.tools.nsc.interpreter.InteractiveReader$.restartSysCalls(InteractiveReader.scala:44)\nscala.tools.nsc.interpreter.InteractiveReader$class.readLine(InteractiveReader.scala:37)\nscala.tools.nsc.interpreter.jline_embedded.InteractiveReader.readLine(JLineReader.scala:28)\nscala.tools.nsc.interpreter.ILoop.readOneLine(ILoop.scala:404)\nscala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:413)\nscala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:923)\nscala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)\nscala.tools.nsc.interp
reter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)\nscala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)\nscala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)\norg.apache.spark.repl.Main$.doMain(Main.scala:76)\norg.apache.spark.repl.Main$.main(Main.scala:56)\norg.apache.spark.repl.Main.main(Main.scala)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)\norg.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)\norg.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)\norg.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)\norg.apache.spark.deploy.SparkSubmit$.main
(SparkSubmit.scala:136)\norg.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)",
"blockedByLock" : "",
"holdingLocks" : [
"Monitor(scala.tools.jline_embedded.internal.NonBlockingInputStream@46248392})"
]
}, {
"threadId" : 26,
"threadName" : "map-output-dispatcher-0",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350285679})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1791280119})" ]
}, {
"threadId" : 27,
"threadName" : "map-output-dispatcher-1",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350285679})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1947378744})" ]
}, {
"threadId" : 28,
"threadName" : "map-output-dispatcher-2",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350285679})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@507507251})" ]
}, {
"threadId" : 29,
"threadName" : "map-output-dispatcher-3",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350285679})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1016408627})" ]
}, {
"threadId" : 30,
"threadName" : "map-output-dispatcher-4",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350285679})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1879219501})" ]
}, {
"threadId" : 31,
"threadName" : "map-output-dispatcher-5",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350285679})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@290509937})" ]
}, {
"threadId" : 32,
"threadName" : "map-output-dispatcher-6",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350285679})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1889468930})" ]
}, {
"threadId" : 33,
"threadName" : "map-output-dispatcher-7",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350285679})",
"holdingLocks" : [
"Lock(java.util.concurrent.ThreadPoolExecutor$Worker@1699637904})" ]
}, {
"threadId" : 47,
"threadName" : "netty-rpc-env-timeout",
"threadState" : "TIMED_WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@977194847})",
"holdingLocks" : [ ]
}, {
"threadId" : 14,
"threadName" : "NonBlockingInputStreamThread",
"threadState" : "WAITING",
"stackTrace" : "java.lang.Object.wait(Native
Method)\nscala.tools.jline_embedded.internal.NonBlockingInputStream.run(NonBlockingInputStream.java:278)\njava.lang.Thread.run(Thread.java:748)",
"blockedByThreadId" : 1,
"blockedByLock" :
"Lock(scala.tools.jline_embedded.internal.NonBlockingInputStream@46248392})",
"holdingLocks" : [ ]
}, {
"threadId" : 2,
"threadName" : "Reference Handler",
"threadState" : "WAITING",
"stackTrace" : "java.lang.Object.wait(Native
Method)\njava.lang.Object.wait(Object.java:502)\njava.lang.ref.Reference.tryHandlePending(Reference.java:191)\njava.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)",
"blockedByLock" : "Lock(java.lang.ref.Reference$Lock@1359433302})",
"holdingLocks" : [ ]
}, {
"threadId" : 35,
"threadName" : "refresh progress",
"threadState" : "TIMED_WAITING",
"stackTrace" : "java.lang.Object.wait(Native
Method)\njava.util.TimerThread.mainLoop(Timer.java:552)\njava.util.TimerThread.run(Timer.java:505)",
"blockedByLock" : "Lock(java.util.TaskQueue@44276328})",
"holdingLocks" : [ ]
}, {
"threadId" : 34,
"threadName" : "RemoteBlock-temp-file-clean-thread",
"threadState" : "TIMED_WAITING",
"stackTrace" : "java.lang.Object.wait(Native
Method)\njava.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)\norg.apache.spark.storage.BlockManager$RemoteBlockTempFileManager.org$apache$spark$storage$BlockManager$RemoteBlockTempFileManager$$keepCleaning(BlockManager.scala:1630)\norg.apache.spark.storage.BlockManager$RemoteBlockTempFileManager$$anon$1.run(BlockManager.scala:1608)",
"blockedByLock" : "Lock(java.lang.ref.ReferenceQueue$Lock@391748181})",
"holdingLocks" : [ ]
}, {
"threadId" : 25,
"threadName" : "rpc-server-3-1",
"threadState" : "RUNNABLE",
"stackTrace" : "sun.nio.ch.KQueueArrayWrapper.kevent0(Native
Method)\nsun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)\nsun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)\nsun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
=> holding
Monitor(sun.nio.ch.KQueueSelectorImpl@2057702496})\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)\nio.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)\nio.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)\nio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)\nio.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)\nio.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" : "",
"holdingLocks" : [
"Monitor(io.netty.channel.nio.SelectedSelectionKeySet@1066929256})",
"Monitor(java.util.Collections$UnmodifiableSet@561426729})",
"Monitor(sun.nio.ch.KQueueSelectorImpl@2057702496})" ]
}, {
"threadId" : 50,
"threadName" : "shuffle-server-5-1",
"threadState" : "RUNNABLE",
"stackTrace" : "sun.nio.ch.KQueueArrayWrapper.kevent0(Native
Method)\nsun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)\nsun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)\nsun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
=> holding
Monitor(sun.nio.ch.KQueueSelectorImpl@1401522546})\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)\nio.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)\nio.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)\nio.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)\nio.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)\nio.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" : "",
"holdingLocks" : [
"Monitor(io.netty.channel.nio.SelectedSelectionKeySet@385972319})",
"Monitor(java.util.Collections$UnmodifiableSet@477937109})",
"Monitor(sun.nio.ch.KQueueSelectorImpl@1401522546})" ]
}, {
"threadId" : 4,
"threadName" : "Signal Dispatcher",
"threadState" : "RUNNABLE",
"stackTrace" : "",
"blockedByLock" : "",
"holdingLocks" : [ ]
}, {
"threadId" : 51,
"threadName" : "Spark Context Cleaner",
"threadState" : "TIMED_WAITING",
"stackTrace" : "java.lang.Object.wait(Native
Method)\njava.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)\norg.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:181)\norg.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1319)\norg.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178)\norg.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)",
"blockedByLock" : "Lock(java.lang.ref.ReferenceQueue$Lock@1739420764})",
"holdingLocks" : [ ]
}, {
"threadId" : 16,
"threadName" : "spark-listener-group-appStatus",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:94)\nscala.util.DynamicVariable.withValue(DynamicVariable.scala:58)\norg.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:83)\norg.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:79)\norg.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1319)\norg.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:78)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1287190987})",
"holdingLocks" : [ ]
}, {
"threadId" : 44,
"threadName" : "spark-listener-group-executorManagement",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:94)\nscala.util.DynamicVariable.withValue(DynamicVariable.scala:58)\norg.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:83)\norg.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:79)\norg.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1319)\norg.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:78)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@943262890})",
"holdingLocks" : [ ]
}, {
"threadId" : 54,
"threadName" : "spark-listener-group-shared",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\norg.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:94)\nscala.util.DynamicVariable.withValue(DynamicVariable.scala:58)\norg.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:83)\norg.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:79)\norg.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1319)\norg.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:78)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@334604425})",
"holdingLocks" : [ ]
}, {
"threadId" : 37,
"threadName" : "SparkUI-37",
"threadState" : "TIMED_WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)\norg.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)\norg.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)\norg.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)\norg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1503479572})",
"holdingLocks" : [ ]
}, {
"threadId" : 38,
"threadName" : "SparkUI-38",
"threadState" : "RUNNABLE",
"stackTrace" : "sun.nio.ch.KQueueArrayWrapper.kevent0(Native
Method)\nsun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)\nsun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)\nsun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
=> holding
Monitor(sun.nio.ch.KQueueSelectorImpl@841741934})\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)\norg.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)\norg.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\norg.spark_project.jetty.u
til.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\norg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" : "",
"holdingLocks" : [ "Monitor(sun.nio.ch.Util$3@873523986})",
"Monitor(java.util.Collections$UnmodifiableSet@1769333189})",
"Monitor(sun.nio.ch.KQueueSelectorImpl@841741934})" ]
}, {
"threadId" : 40,
"threadName" :
"SparkUI-40-acceptor-0@34929380-Spark@3a557b62{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}",
"threadState" : "RUNNABLE",
"stackTrace" : "sun.nio.ch.ServerSocketChannelImpl.accept0(Native
Method)\nsun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)\nsun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
=> holding
Monitor(java.lang.Object@1134240909})\norg.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:371)\norg.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)\norg.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\norg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" : "",
"holdingLocks" : [ "Monitor(java.lang.Object@1134240909})" ]
}, {
"threadId" : 43,
"threadName" : "SparkUI-43",
"threadState" : "RUNNABLE",
"stackTrace" : "sun.management.ThreadImpl.dumpThreads0(Native
Method)\nsun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)\norg.apache.spark.util.Utils$.getThreadDump(Utils.scala:2170)\norg.apache.spark.SparkContext.getExecutorThreadDump(SparkContext.scala:596)\norg.apache.spark.status.api.v1.AbstractApplicationResource$$anonfun$threadDump$1$$anonfun$apply$1.apply(OneApplicationResource.scala:66)\norg.apache.spark.status.api.v1.AbstractApplicationResource$$anonfun$threadDump$1$$anonfun$apply$1.apply(OneApplicationResource.scala:65)\nscala.Option.flatMap(Option.scala:171)\norg.apache.spark.status.api.v1.AbstractApplicationResource$$anonfun$threadDump$1.apply(OneApplicationResource.scala:65)\norg.apache.spark.status.api.v1.AbstractApplicationResource$$anonfun$threadDump$1.apply(OneApplicationResource.scala:58)\norg.apache.spark.status.api.v1.BaseAppResource$$anonfun$withUI$1.apply(ApiRootResource.scala:139)\norg.apache.spark.status.api.v1.BaseAppResource$$anonfun$withUI
$1.apply(ApiRootResource.scala:134)\norg.apache.spark.ui.SparkUI.withSparkUI(SparkUI.scala:106)\norg.apache.spark.status.api.v1.BaseAppResource$class.withUI(ApiRootResource.scala:134)\norg.apache.spark.status.api.v1.AbstractApplicationResource.withUI(OneApplicationResource.scala:32)\norg.apache.spark.status.api.v1.AbstractApplicationResource.threadDump(OneApplicationResource.scala:58)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:498)\norg.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)\norg.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)\norg.glassfish.jersey.server.model.internal.AbstractJavaRe
sourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)\norg.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)\norg.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)\norg.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)\norg.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)\norg.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)\norg.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)\norg.glassfish.jersey.internal.Errors$1.call(Errors.java:271)\norg.glassfish.jersey.internal.Errors$1.call(Errors.java:267)\norg.glassfish.jersey.internal.Errors.process(Errors.java:315)\norg.glassfish.jersey.internal.Errors.process(Errors.java:297)\norg.glassfish.jersey
.internal.Errors.process(Errors.java:267)\norg.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)\norg.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)\norg.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)\norg.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)\norg.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)\norg.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)\norg.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)\norg.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)\norg.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)\norg.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)\norg.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\norg.spark_project.jetty.servlet.ServletHandler.doScope(Servl
etHandler.java:512)\norg.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\norg.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\norg.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)\norg.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\norg.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\norg.spark_project.jetty.server.Server.handle(Server.java:534)\norg.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:320)\norg.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\norg.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\norg.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108)\norg.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\norg.spark_project.jetty.util.thread.strategy
.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\norg.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\norg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" : "",
"holdingLocks" : [ ]
}, {
"threadId" : 67,
"threadName" : "SparkUI-67",
"threadState" : "RUNNABLE",
"stackTrace" : "sun.nio.ch.KQueueArrayWrapper.kevent0(Native
Method)\nsun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)\nsun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)\nsun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
=> holding
Monitor(sun.nio.ch.KQueueSelectorImpl@1837806480})\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)\norg.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)\norg.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\norg.spark_project.jetty.
util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\norg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" : "",
"holdingLocks" : [ "Monitor(sun.nio.ch.Util$3@881415814})",
"Monitor(java.util.Collections$UnmodifiableSet@62050480})",
"Monitor(sun.nio.ch.KQueueSelectorImpl@1837806480})" ]
}, {
"threadId" : 68,
"threadName" : "SparkUI-68",
"threadState" : "RUNNABLE",
"stackTrace" : "sun.nio.ch.KQueueArrayWrapper.kevent0(Native
Method)\nsun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)\nsun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)\nsun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
=> holding
Monitor(sun.nio.ch.KQueueSelectorImpl@223607814})\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)\norg.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)\norg.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\norg.spark_project.jetty.u
til.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\norg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" : "",
"holdingLocks" : [ "Monitor(sun.nio.ch.Util$3@543145185})",
"Monitor(java.util.Collections$UnmodifiableSet@897441546})",
"Monitor(sun.nio.ch.KQueueSelectorImpl@223607814})" ]
}, {
"threadId" : 71,
"threadName" : "SparkUI-71",
"threadState" : "TIMED_WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)\norg.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)\norg.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)\norg.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)\norg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1503479572})",
"holdingLocks" : [ ]
}, {
"threadId" : 77,
"threadName" : "SparkUI-77",
"threadState" : "TIMED_WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)\norg.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)\norg.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)\norg.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)\norg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1503479572})",
"holdingLocks" : [ ]
}, {
"threadId" : 78,
"threadName" : "SparkUI-78",
"threadState" : "RUNNABLE",
"stackTrace" : "sun.nio.ch.KQueueArrayWrapper.kevent0(Native
Method)\nsun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)\nsun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)\nsun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
=> holding
Monitor(sun.nio.ch.KQueueSelectorImpl@403077801})\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)\nsun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)\norg.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)\norg.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\norg.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\norg.spark_project.jetty.u
til.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\norg.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" : "",
"holdingLocks" : [ "Monitor(sun.nio.ch.Util$3@261312406})",
"Monitor(java.util.Collections$UnmodifiableSet@852901260})",
"Monitor(sun.nio.ch.KQueueSelectorImpl@403077801})" ]
}, {
"threadId" : 72,
"threadName" : "SparkUI-JettyScheduler",
"threadState" : "TIMED_WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)\njava.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1587346642})",
"holdingLocks" : [ ]
}, {
"threadId" : 63,
"threadName" : "task-result-getter-0",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@537563105})",
"holdingLocks" : [ ]
}, {
"threadId" : 64,
"threadName" : "task-result-getter-1",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@537563105})",
"holdingLocks" : [ ]
}, {
"threadId" : 65,
"threadName" : "task-result-getter-2",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@537563105})",
"holdingLocks" : [ ]
}, {
"threadId" : 66,
"threadName" : "task-result-getter-3",
"threadState" : "WAITING",
"stackTrace" : "sun.misc.Unsafe.park(Native
Method)\njava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\njava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)\njava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\njava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
"blockedByLock" :
"Lock(java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@537563105})",
"holdingLocks" : [ ]
}, {
"threadId" : 46,
"threadName" : "Timer-0",
"threadState" : "WAITING",
"stackTrace" : "java.lang.Object.wait(Native
Method)\njava.lang.Object.wait(Object.java:502)\njava.util.TimerThread.mainLoop(Timer.java:526)\njava.util.TimerThread.run(Timer.java:505)",
"blockedByLock" : "Lock(java.util.TaskQueue@635634547})",
"holdingLocks" : [ ]
} ]
```
</details>
## How was this patch tested?
It was tested manually.
Old executor page with thread dumps:
<img width="1632" alt="screen shot 2018-02-01 at 14 31 19"
src="https://user-images.githubusercontent.com/2017933/35682124-e2ec5d96-075f-11e8-9713-a502e12d05c2.png";>
New api:
<img width="1669" alt="screen shot 2018-02-01 at 14 31 56"
src="https://user-images.githubusercontent.com/2017933/35682149-f75b80d6-075f-11e8-95b0-c75d048f0b04.png";>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/attilapiros/spark SPARK-23235
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20474.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20474
----
commit 5302b868e80372614aab41acdb20d3935cf85946
Author: âattilapirosâ <piros.attila.zsolt@...>
Date: 2018-01-30T20:11:18Z
Initial commit
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/2017933/35682124-e2ec5d96-075f-11e8-9713-a502e12d05c2.png"](https://user-images.githubusercontent.com/2017933/35682124-e2ec5d96-075f-11e8-9713-a502e12d05c2.png"){kind=link}
![https://user-images.githubusercontent.com/2017933/35682149-f75b80d6-075f-11e8-95b0-c75d048f0b04.png"](https://user-images.githubusercontent.com/2017933/35682149-f75b80d6-075f-11e8-95b0-c75d048f0b04.png"){kind=link}