felipepessoto opened a new pull request, #13563:
URL: https://github.com/apache/gluten/pull/13563
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after
updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the
corresponding issue
when the PR is merged.
-->
**Layer 3 of 3**, split from #13133. Depends on #13561 and its prerequisite
#13560; these references do not close or supersede those PRs.
[Review only this layer (four files, one
commit)](https://github.com/felipepessoto/gluten/compare/d2d3b3756d8a3db7ed076721f75db40c36e36e89...3ff29f81fdc03e7288a39e5dff50551926fd8390).
The head is based directly on layer 2 at
`d2d3b3756d8a3db7ed076721f75db40c36e36e89`. This upstream PR targets `main`
because the prerequisite heads are fork branches, so its ordinary diff includes
the prerequisite layers until they land. No native GitHub stack is registered.
This layer wraps the allocation listener inside `VeloxMemoryManager` with
the existing `wrapJavaException` transport before both Velox arbitration and
Arrow-backed allocation paths use it. The decorator owns its delegate and
forwards current/peak byte metrics unchanged. The runtime/system wrapper
prevents TRY suppression; standard nesting lets direct JNI unreserve calls
recover the original Java throwable without making core JNI depend on Velox.
Adds three native allocator tests and two real-JVM reservation/unreserve
tests while retaining all inherited exception-identity, Spark-metadata,
lazy-output, and strict Delta tests. Completes the memory-callback contract in
`NewToGluten.md`.
No new changes to core JNI, iterator/wrapper implementation, POM wiring,
Delta suites/baselines, native cast translation (#13135), or partition
materialization (#13502). Hybrid Java/C++ stack traces (#9642) remain
explicitly out of scope.
The final aggregate tree is exactly
`eb9ffaec1be46be1017835ad4a3b12737fa7b5fd`, the original source integrated onto
the pinned main base `e63d43ef4f677c860b2a3c27f768572d46725a8c`.
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
Built native first using isolated copies of the layer-2 caches and the exact
IBM/velox `dft-2026_11_05` pin (`eddeec2f17fcb4bf750f2c24bb82232e690e2c5d`).
Rebuilt the changed memory-manager object, native test executable, and backend
JNI library with parallelism capped at four; no duplicate Velox dependency
rebuild.
| Configuration | Executed cases | Result |
| --- | ---: | --- |
| Full `VeloxJavaExceptionTest` via CTest | 8 | Passed |
| Spark 3.5 / Scala 2.12: `GlutenJniExceptionSuite` +
`GlutenDeltaNativeConstraintSuite` | 8 + 6 | Passed |
| Spark 4.1 / Scala 2.13: `GlutenJniExceptionSuite` +
`GlutenDeltaNativeConstraintSuite` | 8 + 6 | Passed |
All 36 cases executed with no failures, errors, or skips. Both JVM
configurations used JDK 17, the `./build/mvn` wrapper, native libraries rebuilt
before Maven, and `-Xcheck:jni` with normal two-library resource/classpath
loading. Reactor builds and scalastyle checks passed for both configurations.
The real-JVM regression control against unchanged layer-2 native libraries
produced 7/8 passes: the native-operator reservation test received
`GlutenException` instead of the original `IllegalStateException`. The same
full eight-case suite passes against this layer's rebuilt libraries. Both Spark
bundles were checked to contain the exact rebuilt native libraries.
Passed `./dev/format-cpp-code.sh` with clang-format 15.0.7,
`./dev/format-scala-code.sh`, per-file CMake formatting, `python3 dev/check.py
format main --fix`, `./.github/workflows/util/check.py header main --fix`, and
`git diff --check`, with no additional source changes.
Existing JNI descriptor/shuffle `populateMetrics` and daemon-thread warnings
were observed; this is not a warning-free claim. The full eight-shard Delta
matrix and 34 targeted upstream Delta cases were not rerun for this layer.
Predecessor #13561 has separately reported Delta shard-6 time-travel failures
under investigation; this PR does not claim that its full CI is green or change
the Delta baseline.
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: GitHub Copilot 1.0.102-0.unstable.r40949665530.gab1eef5 (GPT-6
Astra)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
