[jira] [Created] (SPARK-54386) JavaEncoderSuite testSimpleEncoders has potential to deadlock

[Asif (Jira)]

Asif created SPARK-54386:
----------------------------

             Summary: JavaEncoderSuite testSimpleEncoders has potential to 
deadlock
                 Key: SPARK-54386
                 URL: https://issues.apache.org/jira/browse/SPARK-54386
             Project: Spark
          Issue Type: Bug
          Components: Connect, SQL
    Affects Versions: 4.1.0, 4.2
            Reporter: Asif


The test JavaEncoderSuite.testSimpleEncoders has potential to cause deadlock to 
the fact that Connect Server process spawned from the RemoteSparkSession, does 
not consume from the server process 's input stream, causing the server 
process's output streams ( output and error) to block while logging , in the 
buffers  which get exhausted . As a result shuffle stage does not proceed, with 
the deadlock in DagSchedulerEvent logging ...

I have changes in my master ( but those are valid changes specific to my PRs) 
and may be that has increased the logging... not sure.. need to look into that 
..

The deadlock is not showing up in stock spark master..

But after debugging and fixing the issue in my branch, the cause definitively 
appears to be unconsumed process outout...

Below is the stacktrace of the Connect Spark Server .. showing the 
deadlock....caused by  the logging thread acquiring the Log4J 's class lock, 
and then getting blocked on writing.. 

please note that even though the thread is shown in the Runnable state, it is 
actually blocked... The. reason for this jvm behaviour is available on web...

 
2025-11-17 11:58:36
Full thread dump OpenJDK 64-Bit Server VM (21.0.6 mixed mode, sharing):
 
Threads class SMR info:
_java_thread_list=0x0000600003202280, length=76, elements={
0x000000011f80d400, 0x000000013000c200, 0x000000010f8a9400, 0x000000013000ca00,
0x000000013000d200, 0x0000000130012c00, 0x0000000130013400, 0x0000000130013c00,
0x000000012f811400, 0x000000012f82ac00, 0x000000011f998c00, 0x000000011fadd000,
0x000000011faf4e00, 0x000000011fae1400, 0x000000011faf3800, 0x000000011faf0c00,
0x000000011faf1400, 0x000000011faf1c00, 0x000000011fade600, 0x000000011fadee00,
0x000000011fadf600, 0x000000011fadfe00, 0x000000010fd39800, 0x000000010fd3a000,
0x000000010fd54e00, 0x000000010fd55600, 0x000000010fd55e00, 0x000000010fd56600,
0x000000010fd5a200, 0x000000010fd5aa00, 0x0000000130706400, 0x000000010fd6f800,
0x000000010fd7e000, 0x000000010fd7a000, 0x000000010fd7f600, 0x000000010fd84c00,
0x000000012aba5400, 0x000000012abd6a00, 0x000000012abfd200, 0x0000000130063600,
0x000000011fb12800, 0x000000011fb11e00, 0x000000011fb17400, 0x000000013070b400,
0x000000011fb19c00, 0x000000012ac0c400, 0x000000012ac15a00, 0x000000013071b400,
0x000000011fb24400, 0x000000011fb24c00, 0x000000012ac12800, 0x000000011fb48400,
0x000000012a01e000, 0x000000012b89c200, 0x000000011fc61c00, 0x000000010f91be00,
0x000000010f91cc00, 0x000000012b968000, 0x000000010f946800, 0x000000010f947000,
0x000000010f94c400, 0x000000010f94cc00, 0x000000010fc79a00, 0x000000010fc7a200,
0x000000010fc7aa00, 0x000000010fc7b200, 0x000000010fc7ba00, 0x000000010fc7c200,
0x000000010e858000, 0x00000001305aa400, 0x000000010e857600, 0x000000012a01d600,
0x000000012bd71c00, 0x000000012bd1c400, 0x000000012c14f600, 0x000000010e873a00
}
 
"main" #1 [6147] prio=5 os_prio=31 cpu=1553.09ms elapsed=45.28s 
tid=0x000000011f80d400 nid=6147 runnable  [0x000000016dca9000]
   java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(java.base@21.0.6/Native Method)
at java.io.FileInputStream.read(java.base@21.0.6/FileInputStream.java:287)
at 
java.io.BufferedInputStream.read1(java.base@21.0.6/BufferedInputStream.java:345)
at 
java.io.BufferedInputStream.implRead(java.base@21.0.6/BufferedInputStream.java:420)
at 
java.io.BufferedInputStream.read(java.base@21.0.6/BufferedInputStream.java:399)
at sun.nio.cs.StreamDecoder.readBytes(java.base@21.0.6/StreamDecoder.java:350)
at sun.nio.cs.StreamDecoder.implRead(java.base@21.0.6/StreamDecoder.java:393)
at sun.nio.cs.StreamDecoder.lockedRead(java.base@21.0.6/StreamDecoder.java:217)
at sun.nio.cs.StreamDecoder.read(java.base@21.0.6/StreamDecoder.java:171)
at java.io.InputStreamReader.read(java.base@21.0.6/InputStreamReader.java:188)
at java.io.BufferedReader.fill(java.base@21.0.6/BufferedReader.java:160)
at java.io.BufferedReader.implReadLine(java.base@21.0.6/BufferedReader.java:370)
at java.io.BufferedReader.readLine(java.base@21.0.6/BufferedReader.java:347)
at java.io.BufferedReader.readLine(java.base@21.0.6/BufferedReader.java:436)
at scala.io.StdIn.readLine(StdIn.scala:30)
at scala.io.StdIn.readLine$(StdIn.scala:30)
at scala.io.StdIn$.readLine(StdIn.scala:241)
at 
org.apache.spark.sql.connect.SimpleSparkConnectService$.main(SimpleSparkConnectService.scala:52)
at 
org.apache.spark.sql.connect.SimpleSparkConnectService.main(SimpleSparkConnectService.scala)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@21.0.6/Native 
Method)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@21.0.6/NativeMethodAccessorImpl.java:75)
at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@21.0.6/DelegatingMethodAccessorImpl.java:52)
at java.lang.reflect.Method.invoke(java.base@21.0.6/Method.java:580)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at 
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1028)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:226)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:95)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1166)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1175)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
 
   Locked ownable synchronizers:
- <0x00000007c17c13c0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000007c329dfa0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000007c329dfd0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
 
"Reference Handler" #9 [29443] daemon prio=10 os_prio=31 cpu=2.22ms 
elapsed=45.27s tid=0x000000013000c200 nid=29443 waiting on condition  
[0x000000016ee22000]
   java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base@21.0.6/Native 
Method)
at 
java.lang.ref.Reference.processPendingReferences(java.base@21.0.6/Reference.java:246)
at 
java.lang.ref.Reference$ReferenceHandler.run(java.base@21.0.6/Reference.java:208)
 
   Locked ownable synchronizers:
- None
 
"Finalizer" #10 [29187] daemon prio=8 os_prio=31 cpu=0.14ms elapsed=45.27s 
tid=0x000000010f8a9400 nid=29187 in Object.wait()  [0x000000016f02e000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait0(java.base@21.0.6/Native Method)
- waiting on <0x00000007c131e0f0> (a java.lang.ref.NativeReferenceQueue$Lock)
at java.lang.Object.wait(java.base@21.0.6/Object.java:366)
at java.lang.Object.wait(java.base@21.0.6/Object.java:339)
at 
java.lang.ref.NativeReferenceQueue.await(java.base@21.0.6/NativeReferenceQueue.java:48)
at 
java.lang.ref.ReferenceQueue.remove0(java.base@21.0.6/ReferenceQueue.java:158)
at 
java.lang.ref.NativeReferenceQueue.remove(java.base@21.0.6/NativeReferenceQueue.java:89)
- locked <0x00000007c131e0f0> (a java.lang.ref.NativeReferenceQueue$Lock)
at 
java.lang.ref.Finalizer$FinalizerThread.run(java.base@21.0.6/Finalizer.java:173)
 
   Locked ownable synchronizers:
- None
 
"Signal Dispatcher" #11 [28675] daemon prio=9 os_prio=31 cpu=0.21ms 
elapsed=45.27s tid=0x000000013000ca00 nid=28675 waiting on condition  
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
- None
 
"Service Thread" #12 [24579] daemon prio=9 os_prio=31 cpu=46.92ms 
elapsed=45.27s tid=0x000000013000d200 nid=24579 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
- None
 
"Monitor Deflation Thread" #13 [28419] daemon prio=9 os_prio=31 cpu=1.99ms 
elapsed=45.27s tid=0x0000000130012c00 nid=28419 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
- None
 
"C2 CompilerThread0" #14 [27907] daemon prio=9 os_prio=31 cpu=2910.70ms 
elapsed=45.27s tid=0x0000000130013400 nid=27907 waiting on condition  
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task
 
   Locked ownable synchronizers:
- None
 
"C1 CompilerThread0" #17 [25347] daemon prio=9 os_prio=31 cpu=909.94ms 
elapsed=45.27s tid=0x0000000130013c00 nid=25347 waiting on condition  
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task
 
   Locked ownable synchronizers:
- None
 
"Common-Cleaner" #18 [27651] daemon prio=8 os_prio=31 cpu=2.50ms elapsed=45.26s 
tid=0x000000012f811400 nid=27651 waiting on condition  [0x000000016fc76000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c131e258> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1852)
at java.lang.ref.ReferenceQueue.await(java.base@21.0.6/ReferenceQueue.java:71)
at 
java.lang.ref.ReferenceQueue.remove0(java.base@21.0.6/ReferenceQueue.java:143)
at java.lang.ref.ReferenceQueue.remove(java.base@21.0.6/ReferenceQueue.java:218)
at jdk.internal.ref.CleanerImpl.run(java.base@21.0.6/CleanerImpl.java:140)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
at 
jdk.internal.misc.InnocuousThread.run(java.base@21.0.6/InnocuousThread.java:186)
 
   Locked ownable synchronizers:
- None
 
"Notification Thread" #19 [25859] daemon prio=9 os_prio=31 cpu=0.03ms 
elapsed=45.25s tid=0x000000012f82ac00 nid=25859 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
- None
 
"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #31 
[34307] daemon prio=5 os_prio=31 cpu=0.17ms elapsed=44.63s 
tid=0x000000011f998c00 nid=34307 waiting on condition  [0x0000000179472000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84430> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at java.lang.ref.ReferenceQueue.await(java.base@21.0.6/ReferenceQueue.java:67)
at 
java.lang.ref.ReferenceQueue.remove0(java.base@21.0.6/ReferenceQueue.java:158)
at java.lang.ref.ReferenceQueue.remove(java.base@21.0.6/ReferenceQueue.java:234)
at 
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4171)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"rpc-boss-3-1" #33 [41219] daemon prio=5 os_prio=31 cpu=3.54ms elapsed=43.95s 
tid=0x000000011fadd000 nid=41219 runnable  [0x000000017967e000]
   java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native Method)
at io.netty.channel.kqueue.Native.keventWait(Native.java:133)
at io.netty.channel.kqueue.KQueueIoHandler.kqueueWait(KQueueIoHandler.java:187)
at io.netty.channel.kqueue.KQueueIoHandler.kqueueWait(KQueueIoHandler.java:179)
at io.netty.channel.kqueue.KQueueIoHandler.run(KQueueIoHandler.java:234)
at 
io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
at 
io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1193)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c2e5c5b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
 
"dispatcher-event-loop-0" #34 [35075] daemon prio=5 os_prio=31 cpu=5.67ms 
elapsed=43.94s tid=0x000000011faf4e00 nid=35075 waiting on condition  
[0x000000017988a000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f84c58> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-event-loop-1" #35 [40963] daemon prio=5 os_prio=31 cpu=13.28ms 
elapsed=43.94s tid=0x000000011fae1400 nid=40963 waiting on condition  
[0x0000000179a96000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f84d98> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-event-loop-2" #36 [35587] daemon prio=5 os_prio=31 cpu=1.62ms 
elapsed=43.94s tid=0x000000011faf3800 nid=35587 waiting on condition  
[0x0000000179ca2000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f8ed58> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-event-loop-3" #37 [40195] daemon prio=5 os_prio=31 cpu=3.08ms 
elapsed=43.94s tid=0x000000011faf0c00 nid=40195 waiting on condition  
[0x0000000179eae000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f8f000> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-event-loop-4" #38 [35843] daemon prio=5 os_prio=31 cpu=0.26ms 
elapsed=43.94s tid=0x000000011faf1400 nid=35843 waiting on condition  
[0x000000017a0ba000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f8f1f8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-event-loop-5" #39 [39427] daemon prio=5 os_prio=31 cpu=0.33ms 
elapsed=43.94s tid=0x000000011faf1c00 nid=39427 waiting on condition  
[0x000000017a2c6000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f8f7e0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-event-loop-6" #40 [39171] daemon prio=5 os_prio=31 cpu=0.12ms 
elapsed=43.94s tid=0x000000011fade600 nid=39171 waiting on condition  
[0x000000017a4d2000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f8f3f0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-event-loop-7" #41 [38659] daemon prio=5 os_prio=31 cpu=0.14ms 
elapsed=43.94s tid=0x000000011fadee00 nid=38659 waiting on condition  
[0x000000017a6de000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f8f9d8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-event-loop-8" #42 [38147] daemon prio=5 os_prio=31 cpu=0.06ms 
elapsed=43.94s tid=0x000000011fadf600 nid=38147 waiting on condition  
[0x000000017a8ea000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f8f5e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-event-loop-9" #43 [37891] daemon prio=5 os_prio=31 cpu=0.07ms 
elapsed=43.94s tid=0x000000011fadfe00 nid=37891 waiting on condition  
[0x000000017aaf6000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f84a80> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f8fbd0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"map-output-dispatcher-0" #44 [37379] daemon prio=5 os_prio=31 cpu=0.03ms 
elapsed=43.94s tid=0x000000010fd39800 nid=37379 waiting on condition  
[0x000000017ad02000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f74ef0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:800)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f75058> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"map-output-dispatcher-1" #45 [36867] daemon prio=5 os_prio=31 cpu=0.01ms 
elapsed=43.94s tid=0x000000010fd3a000 nid=36867 waiting on condition  
[0x000000017af0e000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f74ef0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:800)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f75de8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"map-output-dispatcher-2" #46 [43523] daemon prio=5 os_prio=31 cpu=0.01ms 
elapsed=43.94s tid=0x000000010fd54e00 nid=43523 waiting on condition  
[0x000000017b11a000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f74ef0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:800)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f757a0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"map-output-dispatcher-3" #47 [65027] daemon prio=5 os_prio=31 cpu=0.01ms 
elapsed=43.94s tid=0x000000010fd55600 nid=65027 waiting on condition  
[0x000000017b326000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f74ef0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:800)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f75568> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"map-output-dispatcher-4" #48 [43779] daemon prio=5 os_prio=31 cpu=0.01ms 
elapsed=43.94s tid=0x000000010fd55e00 nid=43779 waiting on condition  
[0x000000017b532000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f74ef0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:800)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f76000> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"map-output-dispatcher-5" #49 [64259] daemon prio=5 os_prio=31 cpu=0.02ms 
elapsed=43.94s tid=0x000000010fd56600 nid=64259 waiting on condition  
[0x000000017b73e000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f74ef0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:800)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f75bd0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"map-output-dispatcher-6" #50 [64003] daemon prio=5 os_prio=31 cpu=0.01ms 
elapsed=43.94s tid=0x000000010fd5a200 nid=64003 waiting on condition  
[0x000000017b94a000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f74ef0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:800)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f752b0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"map-output-dispatcher-7" #51 [63747] daemon prio=5 os_prio=31 cpu=0.01ms 
elapsed=43.94s tid=0x000000010fd5aa00 nid=63747 waiting on condition  
[0x000000017bb56000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f74ef0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:800)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1f759b8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-BlockManagerMaster" #52 [44803] daemon prio=5 os_prio=31 cpu=2.75ms 
elapsed=43.92s tid=0x0000000130706400 nid=44803 waiting on condition  
[0x000000017bd62000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2c000f8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c2c001f0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dispatcher-BlockManagerEndpoint1" #53 [63235] daemon prio=5 os_prio=31 
cpu=0.14ms elapsed=43.91s tid=0x000000010fd6f800 nid=63235 waiting on condition 
 [0x000000017bf6e000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2c0ebe8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c2c0ece0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"RemoteBlock-temp-file-clean-thread" #54 [62979] daemon prio=5 os_prio=31 
cpu=3.68ms elapsed=43.90s tid=0x000000010fd7e000 nid=62979 waiting on condition 
 [0x000000017c17a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c335e458> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1852)
at java.lang.ref.ReferenceQueue.await(java.base@21.0.6/ReferenceQueue.java:71)
at 
java.lang.ref.ReferenceQueue.remove0(java.base@21.0.6/ReferenceQueue.java:143)
at java.lang.ref.ReferenceQueue.remove(java.base@21.0.6/ReferenceQueue.java:218)
at 
org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager.org$apache$spark$storage$BlockManager$RemoteBlockDownloadFileManager$$keepCleaning(BlockManager.scala:2297)
at 
org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager$$anon$2.run(BlockManager.scala:2265)
 
   Locked ownable synchronizers:
- None
 
"dispatcher-HeartbeatReceiver" #56 [45571] daemon prio=5 os_prio=31 cpu=2.28ms 
elapsed=43.89s tid=0x000000010fd7a000 nid=45571 waiting on condition  
[0x000000017c386000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2c0eda8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c2c0eea0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"heartbeat-receiver-event-loop-thread" #57 [46083] daemon prio=5 os_prio=31 
cpu=4.55ms elapsed=43.89s tid=0x000000010fd7f600 nid=46083 waiting on condition 
 [0x000000017c592000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1800120> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21.0.6/AbstractQueuedSynchronizer.java:1763)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:1182)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:899)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"netty-rpc-env-timeout" #58 [46339] daemon prio=5 os_prio=31 cpu=0.39ms 
elapsed=43.89s tid=0x000000010fd84c00 nid=46339 waiting on condition  
[0x000000017c79e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c208ce38> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21.0.6/AbstractQueuedSynchronizer.java:1763)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:1182)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:899)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"grpc-nio-boss-ELG-5-1" #59 [46595] daemon prio=5 os_prio=31 cpu=14.97ms 
elapsed=43.84s tid=0x000000012aba5400 nid=46595 runnable  [0x000000017c9aa000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueue.poll(java.base@21.0.6/Native Method)
at 
sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@21.0.6/KQueueSelectorImpl.java:125)
at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@21.0.6/SelectorImpl.java:130)
- locked <0x00000007c1802ca8> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000007c1803cd0> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@21.0.6/SelectorImpl.java:142)
at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at io.netty.channel.nio.NioIoHandler.select(NioIoHandler.java:657)
at io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:434)
at 
io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
at 
io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1193)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1801910> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
 
"dispatcher-org.apache.spark.internal.plugin.PluginEndpoint" #61 [47107] daemon 
prio=5 os_prio=31 cpu=0.12ms elapsed=43.83s tid=0x000000012abd6a00 nid=47107 
waiting on condition  [0x000000017cbb6000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2766b50> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:70)
at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c2766c48> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"dag-scheduler-event-loop" #62 [62211] daemon prio=5 os_prio=31 cpu=75.26ms 
elapsed=43.79s tid=0x000000012abfd200 nid=62211 waiting for monitor entry  
[0x000000017cdc0000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.logging.log4j.core.appender.OutputStreamManager.writeBytes(OutputStreamManager.java:365)
- waiting to lock <0x00000007c1983c40> (a 
org.apache.logging.log4j.core.appender.OutputStreamManager)
at 
org.apache.logging.log4j.core.layout.TextEncoderHelper.writeEncodedText(TextEncoderHelper.java:101)
at 
org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:66)
at 
org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:67)
at 
org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:31)
at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:240)
at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:227)
at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
at 
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
at 
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
at 
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:714)
at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:672)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:648)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:584)
at 
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
at org.apache.logging.log4j.core.Logger.log(Logger.java:187)
at 
org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2904)
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2857)
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2839)
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2623)
at 
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2371)
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:173)
at org.apache.spark.internal.Logging.$anonfun$logInfo$1(Logging.scala:187)
at 
org.apache.spark.internal.Logging$$Lambda/0x00000070014d7b68.apply$mcV$sp(Unknown
 Source)
at org.apache.spark.internal.Logging.withLogContext(Logging.scala:167)
at org.apache.spark.internal.Logging.withLogContext$(Logging.scala:158)
at 
org.apache.spark.scheduler.DAGScheduler.withLogContext(DAGScheduler.scala:124)
at org.apache.spark.internal.Logging.logInfo(Logging.scala:187)
at org.apache.spark.internal.Logging.logInfo$(Logging.scala:184)
at org.apache.spark.scheduler.DAGScheduler.logInfo(DAGScheduler.scala:124)
at 
org.apache.spark.scheduler.DAGScheduler.markStageAsFinished(DAGScheduler.scala:2935)
at 
org.apache.spark.scheduler.DAGScheduler.processShuffleMapStageCompletion(DAGScheduler.scala:2641)
at 
org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:2094)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3254)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3194)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3183)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:50)
 
   Locked ownable synchronizers:
- None
 
"driver-heartbeater" #63 [47875] daemon prio=5 os_prio=31 cpu=3.45ms 
elapsed=43.76s tid=0x0000000130063600 nid=47875 waiting on condition  
[0x000000017cfce000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2766dd8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21.0.6/AbstractQueuedSynchronizer.java:1763)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:1182)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:899)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"executor-kill-mark-cleanup" #64 [61955] daemon prio=5 os_prio=31 cpu=0.44ms 
elapsed=43.75s tid=0x000000011fb12800 nid=61955 waiting on condition  
[0x000000017d1da000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c276adf8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21.0.6/AbstractQueuedSynchronizer.java:1763)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:1182)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:899)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"executor-heartbeater" #65 [48643] daemon prio=5 os_prio=31 cpu=16.69ms 
elapsed=43.75s tid=0x000000011fb11e00 nid=48643 waiting on condition  
[0x000000017d3e6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1864390> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21.0.6/AbstractQueuedSynchronizer.java:1763)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:1182)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:899)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"files-client-7-1" #66 [61699] daemon prio=5 os_prio=31 cpu=57.73ms 
elapsed=43.74s tid=0x000000011fb17400 nid=61699 runnable  [0x000000017d5f2000]
   java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native Method)
at io.netty.channel.kqueue.Native.keventWait(Native.java:133)
at io.netty.channel.kqueue.KQueueIoHandler.kqueueWait(KQueueIoHandler.java:187)
at io.netty.channel.kqueue.KQueueIoHandler.kqueueWait(KQueueIoHandler.java:179)
at io.netty.channel.kqueue.KQueueIoHandler.run(KQueueIoHandler.java:234)
at 
io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
at 
io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1193)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1867560> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
 
"rpc-server-4-1" #67 [49411] daemon prio=5 os_prio=31 cpu=44.33ms 
elapsed=43.74s tid=0x000000013070b400 nid=49411 runnable  [0x000000017d7fe000]
   java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native Method)
at io.netty.channel.kqueue.Native.keventWait(Native.java:133)
at io.netty.channel.kqueue.KQueueIoHandler.kqueueWait(KQueueIoHandler.java:187)
at io.netty.channel.kqueue.KQueueIoHandler.kqueueWait(KQueueIoHandler.java:179)
at io.netty.channel.kqueue.KQueueIoHandler.run(KQueueIoHandler.java:234)
at 
io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
at 
io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1193)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c20f9508> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
 
"process reaper" #68 [49667] daemon prio=10 os_prio=31 cpu=1.59ms 
elapsed=43.68s tid=0x000000011fb19c00 nid=49667 waiting on condition  
[0x000000016fec2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1ee7f88> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
at 
jdk.internal.misc.InnocuousThread.run(java.base@21.0.6/InnocuousThread.java:186)
 
   Locked ownable synchronizers:
- None
 
"shuffle-boss-9-1" #71 [61195] daemon prio=5 os_prio=31 cpu=1.73ms 
elapsed=43.66s tid=0x000000012ac0c400 nid=61195 runnable  [0x000000017da0a000]
   java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native Method)
at io.netty.channel.kqueue.Native.keventWait(Native.java:133)
at io.netty.channel.kqueue.KQueueIoHandler.kqueueWait(KQueueIoHandler.java:187)
at io.netty.channel.kqueue.KQueueIoHandler.kqueueWait(KQueueIoHandler.java:179)
at io.netty.channel.kqueue.KQueueIoHandler.run(KQueueIoHandler.java:234)
at 
io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
at 
io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1193)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c18c6490> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
 
"Spark Context Cleaner" #72 [60675] daemon prio=5 os_prio=31 cpu=23.76ms 
elapsed=43.61s tid=0x000000012ac15a00 nid=60675 waiting on condition  
[0x000000017dc16000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c335c778> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1852)
at java.lang.ref.ReferenceQueue.await(java.base@21.0.6/ReferenceQueue.java:71)
at 
java.lang.ref.ReferenceQueue.remove0(java.base@21.0.6/ReferenceQueue.java:143)
at java.lang.ref.ReferenceQueue.remove(java.base@21.0.6/ReferenceQueue.java:218)
at 
org.apache.spark.ContextCleaner.$anonfun$keepCleaning$1(ContextCleaner.scala:192)
at 
org.apache.spark.ContextCleaner$$Lambda/0x000000700175ea50.apply$mcV$sp(Unknown 
Source)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1277)
at 
org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:190)
at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:80)
 
   Locked ownable synchronizers:
- None
 
"context-cleaner-periodic-gc" #73 [60163] daemon prio=5 os_prio=31 cpu=0.03ms 
elapsed=43.61s tid=0x000000013071b400 nid=60163 waiting on condition  
[0x000000017de22000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2c0ef48> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21.0.6/AbstractQueuedSynchronizer.java:1763)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:1182)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:899)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"spark-listener-group-appStatus" #32 [59651] daemon prio=5 os_prio=31 
cpu=47.82ms elapsed=43.61s tid=0x000000011fb24400 nid=59651 waiting on 
condition  [0x000000017e02e000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f72e88> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:111)
at 
org.apache.spark.scheduler.AsyncEventQueue$$Lambda/0x0000007001760690.apply$mcJ$sp(Unknown
 Source)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
at 
org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:101)
at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:97)
at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2$$Lambda/0x00000070017603c0.apply$mcV$sp(Unknown
 Source)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1277)
at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:97)
 
   Locked ownable synchronizers:
- None
 
"spark-listener-group-executorManagement" #55 [59139] daemon prio=5 os_prio=31 
cpu=4.62ms elapsed=43.61s tid=0x000000011fb24c00 nid=59139 waiting on condition 
 [0x000000017e23a000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c275f708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:111)
at 
org.apache.spark.scheduler.AsyncEventQueue$$Lambda/0x0000007001760690.apply$mcJ$sp(Unknown
 Source)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
at 
org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:101)
at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:97)
at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2$$Lambda/0x00000070017603c0.apply$mcV$sp(Unknown
 Source)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1277)
at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:97)
 
   Locked ownable synchronizers:
- None
 
"spark-listener-group-shared" #60 [50179] daemon prio=5 os_prio=31 cpu=7.03ms 
elapsed=43.61s tid=0x000000012ac12800 nid=50179 waiting on condition  
[0x000000017e446000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2760698> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:111)
at 
org.apache.spark.scheduler.AsyncEventQueue$$Lambda/0x0000007001760690.apply$mcJ$sp(Unknown
 Source)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
at 
org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:101)
at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:97)
at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2$$Lambda/0x00000070017603c0.apply$mcV$sp(Unknown
 Source)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1277)
at 
org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:97)
 
   Locked ownable synchronizers:
- None
 
"element-tracking-store-worker" #74 [50435] daemon prio=5 os_prio=31 cpu=6.03ms 
elapsed=43.60s tid=0x000000011fb48400 nid=50435 waiting on condition  
[0x000000017e652000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1f73cf8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"grpc-nio-worker-ELG-6-1" #75 [50691] daemon prio=5 os_prio=31 cpu=107.47ms 
elapsed=42.48s tid=0x000000012a01e000 nid=50691 runnable  [0x000000017e85e000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueue.poll(java.base@21.0.6/Native Method)
at 
sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@21.0.6/KQueueSelectorImpl.java:125)
at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@21.0.6/SelectorImpl.java:130)
- locked <0x00000007c1826910> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000007c1827938> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@21.0.6/SelectorImpl.java:142)
at 
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at io.netty.channel.nio.NioIoHandler.select(NioIoHandler.java:657)
at io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:434)
at 
io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
at 
io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1193)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1825588> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
 
"grpc-default-executor-0" #76 [50947] daemon prio=5 os_prio=31 cpu=223.83ms 
elapsed=40.26s tid=0x000000012b89c200 nid=50947 waiting on condition  
[0x000000017ea6a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1828380> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"pool-30-thread-1" #77 [57603] prio=5 os_prio=31 cpu=2.49ms elapsed=40.24s 
tid=0x000000011fc61c00 nid=57603 waiting on condition  [0x000000017ec76000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c3140f88> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21.0.6/AbstractQueuedSynchronizer.java:1763)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:1182)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:899)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Cleaner-0" #78 [57347] daemon prio=8 os_prio=31 cpu=0.04ms elapsed=40.14s 
tid=0x000000010f91be00 nid=57347 waiting on condition  [0x000000017ee82000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c3361f28> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1852)
at java.lang.ref.ReferenceQueue.await(java.base@21.0.6/ReferenceQueue.java:71)
at 
java.lang.ref.ReferenceQueue.remove0(java.base@21.0.6/ReferenceQueue.java:143)
at java.lang.ref.ReferenceQueue.remove(java.base@21.0.6/ReferenceQueue.java:218)
at jdk.internal.ref.CleanerImpl.run(java.base@21.0.6/CleanerImpl.java:140)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
at 
jdk.internal.misc.InnocuousThread.run(java.base@21.0.6/InnocuousThread.java:186)
 
   Locked ownable synchronizers:
- None
 
"grpc-default-executor-1" #79 [23815] daemon prio=5 os_prio=31 cpu=161.58ms 
elapsed=39.89s tid=0x000000010f91cc00 nid=23815 waiting on condition  
[0x000000017f08e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c1828380> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"pool-31-thread-1" #81 [56835] prio=5 os_prio=31 cpu=2.46ms elapsed=39.43s 
tid=0x000000012b968000 nid=56835 waiting on condition  [0x000000017f29a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c3365838> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21.0.6/AbstractQueuedSynchronizer.java:1763)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:1182)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@21.0.6/ScheduledThreadPoolExecutor.java:899)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 0.0 in stage 0.0 (TID 0)" #85 [51971] 
daemon prio=5 os_prio=31 cpu=33.13ms elapsed=38.52s tid=0x000000010f946800 
nid=51971 waiting on condition  [0x000000017f8be000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 1.0 in stage 0.0 (TID 1)" #86 [55299] 
daemon prio=5 os_prio=31 cpu=34.34ms elapsed=38.52s tid=0x000000010f947000 
nid=55299 waiting on condition  [0x000000017faca000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 2.0 in stage 0.0 (TID 2)" #87 [54787] 
daemon prio=5 os_prio=31 cpu=44.52ms elapsed=38.52s tid=0x000000010f94c400 
nid=54787 waiting on condition  [0x000000017fcd6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 3.0 in stage 0.0 (TID 3)" #88 [54531] 
daemon prio=5 os_prio=31 cpu=43.28ms elapsed=38.52s tid=0x000000010f94cc00 
nid=54531 waiting on condition  [0x000000017fee2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 4.0 in stage 0.0 (TID 4)" #89 [52739] 
daemon prio=5 os_prio=31 cpu=63.84ms elapsed=38.52s tid=0x000000010fc79a00 
nid=52739 waiting on condition  [0x000000030c206000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 5.0 in stage 0.0 (TID 5)" #90 [54275] 
daemon prio=5 os_prio=31 cpu=66.52ms elapsed=38.52s tid=0x000000010fc7a200 
nid=54275 waiting on condition  [0x000000030c412000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 0.0 in stage 1.0 (TID 10)" #91 [53763] 
daemon prio=5 os_prio=31 cpu=49.09ms elapsed=38.52s tid=0x000000010fc7aa00 
nid=53763 waiting on condition  [0x000000030c61e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 2.0 in stage 1.0 (TID 12)" #92 [53507] 
daemon prio=5 os_prio=31 cpu=42.68ms elapsed=38.52s tid=0x000000010fc7b200 
nid=53507 waiting on condition  [0x000000030c82a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 8.0 in stage 0.0 (TID 8)" #93 [65795] 
daemon prio=5 os_prio=31 cpu=30.79ms elapsed=38.52s tid=0x000000010fc7ba00 
nid=65795 waiting on condition  [0x000000030ca36000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Executor task launch worker for task 1.0 in stage 1.0 (TID 11)" #94 [66307] 
daemon prio=5 os_prio=31 cpu=56.59ms elapsed=38.52s tid=0x000000010fc7c200 
nid=66307 waiting on condition  [0x000000030cc42000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c27673c0> (a 
java.util.concurrent.SynchronousQueue$Transferer)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:410)
at 
java.util.concurrent.LinkedTransferQueue$DualNode.await(java.base@21.0.6/LinkedTransferQueue.java:452)
at 
java.util.concurrent.SynchronousQueue$Transferer.xferLifo(java.base@21.0.6/SynchronousQueue.java:194)
at 
java.util.concurrent.SynchronousQueue.xfer(java.base@21.0.6/SynchronousQueue.java:235)
at 
java.util.concurrent.SynchronousQueue.poll(java.base@21.0.6/SynchronousQueue.java:338)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"task-result-getter-0" #121 [66863] daemon prio=5 os_prio=31 cpu=3.28ms 
elapsed=38.30s tid=0x000000010e858000 nid=66863 waiting on condition  
[0x000000030ce4e000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2765b20> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"task-result-getter-1" #122 [87043] daemon prio=5 os_prio=31 cpu=2.88ms 
elapsed=38.30s tid=0x00000001305aa400 nid=87043 runnable  [0x000000030d058000]
   java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(java.base@21.0.6/Native Method)
at java.io.FileOutputStream.write(java.base@21.0.6/FileOutputStream.java:367)
at 
java.io.BufferedOutputStream.flushBuffer(java.base@21.0.6/BufferedOutputStream.java:125)
at 
java.io.BufferedOutputStream.implFlush(java.base@21.0.6/BufferedOutputStream.java:252)
at 
java.io.BufferedOutputStream.flush(java.base@21.0.6/BufferedOutputStream.java:240)
at java.io.PrintStream.implWrite(java.base@21.0.6/PrintStream.java:645)
at java.io.PrintStream.write(java.base@21.0.6/PrintStream.java:623)
at 
org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.java:53)
at 
org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:263)
- locked <0x00000007c1983c40> (a 
org.apache.logging.log4j.core.appender.OutputStreamManager)
at 
org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:296)
- locked <0x00000007c1983c40> (a 
org.apache.logging.log4j.core.appender.OutputStreamManager)
at 
org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:307)
- locked <0x00000007c1983c40> (a 
org.apache.logging.log4j.core.appender.OutputStreamManager)
at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:229)
at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
at 
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
at 
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
at 
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:714)
at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:672)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:648)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:584)
at 
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
at org.apache.logging.log4j.core.Logger.log(Logger.java:187)
at 
org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2904)
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2857)
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2839)
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2623)
at 
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2371)
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:173)
at org.apache.spark.internal.Logging.$anonfun$logInfo$1(Logging.scala:187)
at 
org.apache.spark.internal.Logging$$Lambda/0x00000070014d7b68.apply$mcV$sp(Unknown
 Source)
at org.apache.spark.internal.Logging.withLogContext(Logging.scala:167)
at org.apache.spark.internal.Logging.withLogContext$(Logging.scala:158)
at 
org.apache.spark.scheduler.TaskSchedulerImpl.withLogContext(TaskSchedulerImpl.scala:83)
at org.apache.spark.internal.Logging.logInfo(Logging.scala:187)
at org.apache.spark.internal.Logging.logInfo$(Logging.scala:184)
at 
org.apache.spark.scheduler.TaskSchedulerImpl.logInfo(TaskSchedulerImpl.scala:83)
at 
org.apache.spark.scheduler.TaskSchedulerImpl.taskSetFinished(TaskSchedulerImpl.scala:359)
- locked <0x00000007c2cce260> (a org.apache.spark.scheduler.TaskSchedulerImpl)
at 
org.apache.spark.scheduler.TaskSetManager.maybeFinishTaskSet(TaskSetManager.scala:601)
at 
org.apache.spark.scheduler.TaskSetManager.handleSuccessfulTask(TaskSetManager.scala:885)
at 
org.apache.spark.scheduler.TaskSchedulerImpl.handleSuccessfulTask(TaskSchedulerImpl.scala:873)
- locked <0x00000007c2cce260> (a org.apache.spark.scheduler.TaskSchedulerImpl)
at 
org.apache.spark.scheduler.TaskResultGetter$$anon$3.$anonfun$run$1(TaskResultGetter.scala:122)
at 
org.apache.spark.scheduler.TaskResultGetter$$anon$3$$Lambda/0x00000070021b9998.apply$mcV$sp(Unknown
 Source)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1925)
at 
org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:64)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1144)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- <0x00000007c1682d88> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000007c1682e68> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000007c444cf38> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"task-result-getter-2" #123 [67331] daemon prio=5 os_prio=31 cpu=2.27ms 
elapsed=38.30s tid=0x000000010e857600 nid=67331 waiting on condition  
[0x000000030d266000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2765b20> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"task-result-getter-3" #124 [86531] daemon prio=5 os_prio=31 cpu=1.48ms 
elapsed=38.30s tid=0x000000012a01d600 nid=86531 waiting on condition  
[0x000000030d472000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c2765b20> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1070)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"SparkConnectExecuteThread_opId=58cbe037-7053-44b7-b62e-f41027f1962c" #130 
[56347] daemon prio=5 os_prio=31 cpu=185.69ms elapsed=38.14s 
tid=0x000000012bd71c00 nid=56347 waiting on condition  [0x000000017f4a4000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c4460318> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.6/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.6/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.6/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.6/AbstractQueuedSynchronizer.java:1712)
at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@21.0.6/LinkedBlockingQueue.java:435)
at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.waitForNextStageToMaterialize(AdaptiveSparkPlanExec.scala:597)
at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$withFinalPlanUpdate$1(AdaptiveSparkPlanExec.scala:410)
at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec$$Lambda/0x00000070022addd8.apply$mcV$sp(Unknown
 Source)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804)
at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:291)
- locked <0x00000007c44607a0> (a java.lang.Object)
at 
org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.doExecute(AdaptiveSparkPlanExec.scala:715)
at 
org.apache.spark.sql.execution.SparkPlan.$anonfun$executeRDD$1(SparkPlan.scala:189)
at 
org.apache.spark.sql.execution.SparkPlan$$Lambda/0x0000007001e8b970.apply(Unknown
 Source)
at scala.util.Try$.apply(Try.scala:217)
at org.apache.spark.util.Utils$.doTryWithCallerStacktrace(Utils.scala:1376)
at org.apache.spark.util.LazyTry.tryT$lzycompute(LazyTry.scala:46)
- locked <0x00000007c44607c0> (a org.apache.spark.util.LazyTry)
at org.apache.spark.util.LazyTry.tryT(LazyTry.scala:46)
at org.apache.spark.util.LazyTry.get(LazyTry.scala:58)
at 
org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:202)
at 
org.apache.spark.sql.execution.SparkPlan$$Lambda/0x0000007001edf6b0.apply(Unknown
 Source)
at 
org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:261)
at 
org.apache.spark.sql.execution.SparkPlan$$Lambda/0x0000007001ee6bb8.apply(Unknown
 Source)
at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:258)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:198)
at 
org.apache.spark.sql.connect.execution.SparkConnectPlanExecution.$anonfun$processAsArrowBatches$3(SparkConnectPlanExecution.scala:212)
at 
org.apache.spark.sql.connect.execution.SparkConnectPlanExecution$$Lambda/0x0000007001eb4f90.apply(Unknown
 Source)
at 
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$8(SQLExecution.scala:176)
at 
org.apache.spark.sql.execution.SQLExecution$$$Lambda/0x0000007001ebc2b8.apply(Unknown
 Source)
at 
org.apache.spark.sql.execution.SQLExecution$.withSessionTagsApplied(SQLExecution.scala:284)
at 
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$7(SQLExecution.scala:138)
at 
org.apache.spark.sql.execution.SQLExecution$$$Lambda/0x0000007001ebc000.apply(Unknown
 Source)
at 
org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94)
at 
org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:112)
at 
org.apache.spark.sql.artifact.ArtifactManager$$Lambda/0x0000007001a98778.apply(Unknown
 Source)
at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:185)
at 
org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:102)
at 
org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:111)
at 
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$6(SQLExecution.scala:138)
at 
org.apache.spark.sql.execution.SQLExecution$$$Lambda/0x0000007001eb7388.apply(Unknown
 Source)
at 
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:307)
at 
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:137)
at 
org.apache.spark.sql.execution.SQLExecution$$$Lambda/0x0000007001eb5240.apply(Unknown
 Source)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804)
at 
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId0(SQLExecution.scala:91)
at 
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:249)
at 
org.apache.spark.sql.connect.execution.SparkConnectPlanExecution.processAsArrowBatches(SparkConnectPlanExecution.scala:211)
at 
org.apache.spark.sql.connect.execution.SparkConnectPlanExecution.handlePlan(SparkConnectPlanExecution.scala:79)
at 
org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1(ExecuteThreadRunner.scala:225)
at 
org.apache.spark.sql.connect.execution.ExecuteThreadRunner.$anonfun$executeInternal$1$adapted(ExecuteThreadRunner.scala:197)
at 
org.apache.spark.sql.connect.execution.ExecuteThreadRunner$$Lambda/0x0000007001bbf5b8.apply(Unknown
 Source)
at 
org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$2(SessionHolder.scala:396)
at 
org.apache.spark.sql.connect.service.SessionHolder$$Lambda/0x0000007001a9a250.apply(Unknown
 Source)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804)
at 
org.apache.spark.sql.connect.service.SessionHolder.$anonfun$withSession$1(SessionHolder.scala:396)
at 
org.apache.spark.sql.connect.service.SessionHolder$$Lambda/0x0000007001a984c8.apply(Unknown
 Source)
at 
org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94)
at 
org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:112)
at 
org.apache.spark.sql.artifact.ArtifactManager$$Lambda/0x0000007001a98778.apply(Unknown
 Source)
at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:185)
at 
org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:102)
at 
org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:111)
at 
org.apache.spark.sql.connect.service.SessionHolder.withSession(SessionHolder.scala:395)
at 
org.apache.spark.sql.connect.execution.ExecuteThreadRunner.executeInternal(ExecuteThreadRunner.scala:197)
at 
org.apache.spark.sql.connect.execution.ExecuteThreadRunner.org$apache$spark$sql$connect$execution$ExecuteThreadRunner$$execute(ExecuteThreadRunner.scala:126)
at 
org.apache.spark.sql.connect.execution.ExecuteThreadRunner$ExecutionThread.run(ExecuteThreadRunner.scala:334)
 
   Locked ownable synchronizers:
- None
 
"SparkConnectGRPCSender_opId=58cbe037-7053-44b7-b62e-f41027f1962c_startIndex=2" 
#133 [56087] daemon prio=5 os_prio=31 cpu=1.11ms elapsed=37.98s 
tid=0x000000012bd1c400 nid=56087 waiting for monitor entry  [0x000000017f6b1000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.logging.log4j.core.appender.OutputStreamManager.writeBytes(OutputStreamManager.java:365)
- waiting to lock <0x00000007c1983c40> (a 
org.apache.logging.log4j.core.appender.OutputStreamManager)
at 
org.apache.logging.log4j.core.layout.TextEncoderHelper.writeEncodedText(TextEncoderHelper.java:101)
at 
org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:66)
at 
org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:67)
at 
org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:31)
at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:240)
at 
org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:227)
at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
at 
org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
at 
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
at 
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
at 
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
at 
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:714)
at 
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:672)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:648)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:584)
at 
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
at org.apache.logging.log4j.core.Logger.log(Logger.java:187)
at 
org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2904)
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2857)
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2839)
at 
org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2623)
at 
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2371)
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:173)
at org.apache.spark.internal.Logging.$anonfun$logInfo$1(Logging.scala:187)
at 
org.apache.spark.internal.Logging$$Lambda/0x00000070014d7b68.apply$mcV$sp(Unknown
 Source)
at org.apache.spark.internal.Logging.withLogContext(Logging.scala:167)
at org.apache.spark.internal.Logging.withLogContext$(Logging.scala:158)
at 
org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender.withLogContext(ExecuteGrpcResponseSender.scala:43)
at org.apache.spark.internal.Logging.logInfo(Logging.scala:187)
at org.apache.spark.internal.Logging.logInfo$(Logging.scala:184)
at 
org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender.logInfo(ExecuteGrpcResponseSender.scala:43)
at 
org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender.execute(ExecuteGrpcResponseSender.scala:331)
at 
org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender$$anon$1.run(ExecuteGrpcResponseSender.scala:102)
 
   Locked ownable synchronizers:
- None
 
"shuffle-exchange-0" #134 [86019] daemon prio=5 os_prio=31 cpu=15.89ms 
elapsed=37.95s tid=0x000000012c14f600 nid=86019 waiting on condition  
[0x000000030d67e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
- parking to wait for  <0x00000007c444d0a8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@21.0.6/LockSupport.java:269)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@21.0.6/AbstractQueuedSynchronizer.java:1763)
at 
java.util.concurrent.LinkedBlockingQueue.poll(java.base@21.0.6/LinkedBlockingQueue.java:460)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@21.0.6/ThreadPoolExecutor.java:1069)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@21.0.6/ThreadPoolExecutor.java:1130)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@21.0.6/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)
 
   Locked ownable synchronizers:
- None
 
"Attach Listener" #137 [68103] daemon prio=9 os_prio=31 cpu=1.00ms 
elapsed=0.13s tid=0x000000010e873a00 nid=68103 waiting on condition  
[0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
- None
 
"G1 Conc#1" os_prio=31 cpu=63.91ms elapsed=44.67s tid=0x000000012f6b8100 
nid=41731 runnable  
 
"GC Thread#8" os_prio=31 cpu=16.39ms elapsed=44.67s tid=0x000000012f7aa200 
nid=41987 runnable  
 
"GC Thread#7" os_prio=31 cpu=28.43ms elapsed=44.67s tid=0x000000012f7a9c90 
nid=33795 runnable  
 
"GC Thread#6" os_prio=31 cpu=13.51ms elapsed=44.67s tid=0x000000012f7a9720 
nid=33539 runnable  
 
"GC Thread#5" os_prio=31 cpu=28.57ms elapsed=45.12s tid=0x000000012f622400 
nid=43011 runnable  
 
"GC Thread#4" os_prio=31 cpu=23.59ms elapsed=45.12s tid=0x000000012f621e90 
nid=43267 runnable  
 
"GC Thread#3" os_prio=31 cpu=33.29ms elapsed=45.12s tid=0x000000012f621920 
nid=32771 runnable  
 
"GC Thread#2" os_prio=31 cpu=23.20ms elapsed=45.12s tid=0x000000012f6213b0 
nid=26627 runnable  
 
"GC Thread#1" os_prio=31 cpu=27.39ms elapsed=45.12s tid=0x000000012f620e40 
nid=26115 runnable  
 
"VM Thread" os_prio=31 cpu=36.00ms elapsed=45.28s tid=0x000000011800a740 
nid=18179 runnable  
 
"VM Periodic Task Thread" os_prio=31 cpu=12.12ms elapsed=45.28s 
tid=0x0000000118007a70 nid=20995 waiting on condition  
 
"G1 Service" os_prio=31 cpu=1.37ms elapsed=45.28s tid=0x000000010f704ba0 
nid=16643 runnable  
 
"G1 Refine#0" os_prio=31 cpu=15.57ms elapsed=45.28s tid=0x000000010f894800 
nid=13059 runnable  
 
"G1 Conc#0" os_prio=31 cpu=64.76ms elapsed=45.28s tid=0x000000010f704100 
nid=12547 runnable  
 
"G1 Main Marker" os_prio=31 cpu=0.75ms elapsed=45.28s tid=0x00000001180042b0 
nid=12035 runnable  
 
"GC Thread#0" os_prio=31 cpu=25.43ms elapsed=45.28s tid=0x000000010f6071f0 
nid=13827 runnable  
 
JNI global refs: 52, weak refs: 4
 
will be opening a PR for the same..



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org