felipepessoto commented on PR #12290:
URL: https://github.com/apache/gluten/pull/12290#issuecomment-4774760943
@rui-mo, digging in, tmp->field() isn't actually returning a negative value,
and I don't think there's a hidden bug here. The field index comes from Spark's
resolved ordinals (BoundReference.ordinal / GetStructField.ordinal), which are
validated and non-negative, and even a bad index couldn't segfault:
RowType::childAt/nameOf are bounds-checked and throw a catchable exception.
I also have to correct my earlier root cause: the spark-test-spark40 crash
I'd blamed on removing this guard turned out to be unrelated. The hs_err shows
a flaky native use-after-free in NativeMemoryManagerJniWrapper.hold during
VeloxColumnarToRow iterator teardown (on the script-transformation feed thread)
— no field-reference frames involved.
So the real fix in this PR is the VELOX_USER_CHECK_NOT_NULL for the
array/map-element case; the bounds check is just cheap, redundant
defense-in-depth.
```
1. Crash signature — executing a non-code address
# SIGSEGV (0xb) at pc=0x00007f39180b95f0, pid=172686, tid=181384
# Problematic frame:
# C 0x00007f39180b95f0 ← unsymbolized: not in any function
(line 54) siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR),
si_addr: 0x00007f39180b95f0
si_addr == pc → the CPU faulted fetching the instruction at 0x7f39180b95f0
(a jump/call to that address), not reading data. SEGV_ACCERR = the page exists
but the access is not permitted.
2. The faulting address is in writable, NON-executable anonymous memory
(line 58) RAX=0x00007f39180b95f0 points into unknown readable memory ...
Memory map (line 3295):
7f3918000000-7f3918d97000 rw-p 00000000 00:00 0 ← pc lives here
The call dispatched through RAX = 0x7f39180b95f0, which lands in an rw-p
(read/write, no x) 00:00 0 (anonymous, not a loaded .so) region — i.e. an
off-heap data buffer. Calling through a pointer that now points at
freed/garbage data instead of code is the textbook signature of a dangling
function pointer / use-after-free.
3. What was running: ColumnarToRow iterator teardown calling the native
memory manager
Current thread (line 28):
"Thread-SparkScriptTransformationWriterThread-Feed" [_thread_in_native]
Java frames (lines 38-43):
NativeMemoryManagerJniWrapper.hold(J)V ← native
memory-manager handle call
VeloxIteratorApi$$Lambda$13357.apply$mcV$sp() ← completion
callback
IteratorsV1$IteratorCompleter.hasNext()
IteratorsV1$PayloadCloser.hasNext() ← iterator
TEARDOWN/close
IteratorsV1$LifeTimeAccumulator.hasNext()
VeloxColumnarToRowExec$$anon$1.hasNext()
The native object is NativeMemoryManager$Impl {0x0000000802debe60} (line
61). So the feed thread, during VeloxColumnarToRow iterator
completion/teardown, calls NativeMemoryManager.hold(handle) and dispatches into
a freed native object.
4. No field-reference code involved
field-ref frames in crashing stack: 0
Zero substrait / FieldReference / toVeloxExpr / childAt frames in the
crashing thread (the only "substrait" strings in the whole file are unrelated
<clinit> classloading events at t=153s).
```
Conclusion: native use-after-free in NativeMemoryManager.hold during
ColumnarToRow teardown — a memory-lifecycle race, unrelated to the
field-reference guard removed in that commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
