yikf commented on PR #12714:
URL: https://github.com/apache/gluten/pull/12714#issuecomment-5204534257
Thanks @zhztheplayer. You're right that in the normal flow the iterator is
closed before the `Runtime` is destroyed — but the ordering guarantee only
holds within the task thread, and script transformation breaks it by closing
the iterator from a second thread.
`resIter.close()` (which calls `hold()`) can be triggered from two places
which are `recycler` and `hasNext`, both funneled through
`IteratorCompleter.tryComplete()` and guarded by a CAS so it runs exactly once:
- the TaskResources recycler (priority 100), run on the task thread at task
completion;
- `IteratorCompleter.hasNext` returning false, run on whatever thread is
driving the iterator.
In the normal single-threaded path both run on the task thread, and
`releaseAll()` executes resources by descending priority, so the iterator
recycler (100) holds before the Runtime (30) releases — hold() always precedes
release().
`SparkScriptTransformationExec.processIterator` starts
`BaseScriptTransformationWriterThread` (a setDaemon(true) "Feed" thread) and
never joins it; that thread drains the Gluten input iterator via
`iter.foreach`. The race:
```
1. Feed thread: hasNext hits end-of-stream → tryComplete() wins the CAS
(marks completed) but gets descheduled before running resIter.close().
2. Task thread: main thread finishes reading the script's stdout → task
completes → the recycler's tryComplete() finds the CAS already taken and
no-ops, so the task thread never holds and proceeds straight to
Runtime.release() → nmm.release(), freeing the native handle.
3. Feed thread: resumes, runs resIter.close() →
ColumnarBatchOutIterator.close0() → hold() on the now-freed handle → SIGSEGV.
```
So `hold()` runs exactly once, but on the feed thread, with no
happens-before against the task thread's `release()`. The hs_err stack matches
— `hold()` under `BaseScriptTransformationWriterThread.run()`.
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f01a868459e, pid=7037, tid=7720
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.6.0.9-0.3.ea.el8)
(17.0.6+9) (build 17.0.6-ea+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.6.0.9-0.3.ea.el8)
(17.0.6-ea+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed
class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C [libgluten.so+0x4fe59e]
Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
#
# Core dump will be written. Default location: Core dumps may be processed
with "/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %d"
(or dumping to /__w/gluten/gluten/gluten-ut/spark40/core.7037)
#
# If you would like to submit a bug report, please visit:
#
https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-17-openjdk
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- S U M M A R Y ------------
Command Line:
-Dlog4j.configurationFile=file:src/test/resources/log4j2.properties
-Dbasedir=/__w/gluten/gluten/gluten-ut/spark40
-Dspark.test.home=/opt/shims/spark40/spark_home/
-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false
-Dio.netty.tryReflectionSetAccessible=true -Dfile.encoding=
UTF-8 org.scalatest.tools.Runner -R
/__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/classes
/__w/gluten/gluten/gluten-ut/spark40/target/scala-2.13/test-classes -l
org.apache.spark.tags.ExtendedSQLTest org.apache.spark.tags.SlowHiveTest
org.apache.gluten.tags.UDFTest org.apache.gluten.tags.SkipTest -w
org.apache.spark.sql.execution -w org.apache.spark.sql.catalyst -w
org.apache.spark.sql.errors -w org.apache.spark.sql.extension -o -u
/__w/gluten/gluten/gluten-ut/spark40/target/surefire-reports/.
Host: AMD EPYC 9V74 80-Core Processor, 4 cores, 15G, CentOS Stream release 8
Time: Thu Aug 6 05:15:16 2026 UTC elapsed time: 22.784636 seconds (0d 0h 0m
22s)
--------------- T H R E A D ---------------
Current thread (0x00007f021007c970): JavaThread
"Thread-SparkScriptTransformationWriterThread-Feed" daemon [_thread_in_native,
id=7720, stack(0x00007f0180ed1000,0x00007f0180fd2000)]
Stack: [0x00007f0180ed1000,0x00007f0180fd2000], sp=0x00007f0180fcff60,
free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
C [libgluten.so+0x4fe59e]
Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_hold+0x1e
j org.apache.gluten.memory.NativeMemoryManager$Impl.hold()V+4
j org.apache.gluten.vectorized.ColumnarBatchOutIterator.close0()V+9
j org.apache.gluten.iterator.ClosableIterator.close()V+13
j
org.apache.gluten.backendsapi.velox.VeloxIteratorApi.$anonfun$genFinalStageIterator$2(Lscala/Function1;Lorg/apache/gluten/metrics/IteratorMetricsJniWrapper;Lorg/apache/gluten/vectorized/ColumnarBatchOutIterator;)V+13
j
org.apache.gluten.backendsapi.velox.VeloxIteratorApi$$Lambda$7598+0x0000000802b4afc8.apply$mcV$sp()V+12
J 18857 c1
org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext()Z (23 bytes)
@ 0x00007f0226a9bed4 [0x00007f0226a9b780+0x0000000000000754]
J 18316 c1 org.apache.gluten.iterator.IteratorsV1$PayloadCloser.hasNext()Z
(14 bytes) @ 0x00007f0225f5a3f4 [0x00007f0225f5a2c0+0x0000000000000134]
j org.apache.gluten.iterator.IteratorsV1$LifeTimeAccumulator.hasNext()Z+4
j org.apache.gluten.execution.VeloxColumnarToRowExec$$anon$1.hasNext()Z+4
J 15005 c2 scala.collection.Iterator$$anon$10.hasNext()Z (76 bytes) @
0x00007f022d0418d0 [0x00007f022d0417c0+0x0000000000000110]
J 18859 c1
org.apache.gluten.iterator.IteratorsV1$InvocationFlowProtection.hasNext()Z (122
bytes) @ 0x00007f0226a9d26c [0x00007f0226a9cc80+0x00000000000005ec]
J 18857 c1
org.apache.gluten.iterator.IteratorsV1$IteratorCompleter.hasNext()Z (23 bytes)
@ 0x00007f0226a9b894 [0x00007f0226a9b780+0x0000000000000114]
J 963 c2 scala.collection.Iterator$$anon$9.hasNext()Z (10 bytes) @
0x00007f022c94b1e8 [0x00007f022c94b1a0+0x0000000000000048]
J 10777 c2 scala.collection.IterableOnceOps.foreach(Lscala/Function1;)V (36
bytes) @ 0x00007f022ce7bf70 [0x00007f022ce7bea0+0x00000000000000d0]
J 6133 c1 scala.collection.AbstractIterator.foreach(Lscala/Function1;)V (6
bytes) @ 0x00007f0225794dec [0x00007f0225794d40+0x00000000000000ac]
j
org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.processRowsWithoutSerde()V+21
j
org.apache.spark.sql.execution.SparkScriptTransformationWriterThread.processRows()V+1
j
org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.$anonfun$run$1(Lorg/apache/spark/sql/execution/BaseScriptTransformationWriterThread;)V+13
j
org.apache.spark.sql.execution.BaseScriptTransformationWriterThread$$Lambda$8729+0x0000000802dd1da8.apply$mcV$sp()V+4
J 14838 c2 scala.runtime.java8.JFunction0$mcV$sp.apply()Ljava/lang/Object;
(10 bytes) @ 0x00007f022d02585c [0x00007f022d025820+0x000000000000003c]
j
org.apache.spark.util.Utils$.logUncaughtExceptions(Lscala/Function0;)Ljava/lang/Object;+1
j
org.apache.spark.sql.execution.BaseScriptTransformationWriterThread.run()V+9
v ~StubRoutines::call_stub
V [libjvm.so+0x843dba] JavaCalls::call_helper(JavaValue*, methodHandle
const&, JavaCallArguments*, JavaThread*)+0x32a
".//spark-test-spark40-group2-test-log/gluten-ut/spark40/hs_err_pid7037.log"
2057L, 177036B
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
