malinjawi opened a new pull request, #12867:
URL: https://github.com/apache/gluten/pull/12867
## What changes were proposed in this pull request?
This is PR B of the Delta deletion-vector performance fix and is stacked on
#12836.
- PR A (#12836) moves JVM deletion-vector materialization from serial driver
planning to Spark executors. That is the regression fix and remains the default
path.
- This PR adds a separately reviewable, opt-in Velox range-read path. Until
#12836 merges, the GitHub diff against `main` contains both commits; the second
commit (`387903012`) is the PR B delta.
When `spark.gluten.sql.columnar.delta.dv.nativeRangeRead.enabled=true`, the
JVM resolves the authoritative absolute deletion-vector URI and passes
`{absolute path, offset, payload size}` to Velox. Velox reads the exact stored
range through `FileHandleFactory` and `BufferedInput`, allowing the existing
split preload and async data-cache paths to overlap and reuse I/O. Inline
deletion vectors continue to use the JVM byte handoff.
The descriptor is additive in Substrait: the existing serialized-byte field
remains available for backward compatibility. The native reader validates the
requested range, Delta length prefix, CRC checksum, Roaring payload, and
cardinality before applying the bitmap.
The feature defaults to `false`. When enabled it takes precedence over
`spark.gluten.sql.columnar.delta.dv.deferPayloadRead.enabled`.
## Why are the changes needed?
#12836 removes the approximately 100-second serial driver barrier seen with
2,461 deletion-vector files on S3. This optional follow-up avoids the remaining
executor-JVM materialization and JNI payload copy, and lets deletion-vector I/O
participate in Velox file-handle caching and split preloading.
On the original combined benchmark image, the four fixed native totals were
21.821s, 22.110s, 21.366s, and 21.674s (median 21.748s), versus 257.976s for
the legacy native path, 76.316s for vanilla Spark, and 128.310s for Gluten
fallback. Query-result digests matched and 103 TPC-DS variants passed. This
demonstrates the combined A+B path; a matched deferred-only S3 arm is still
required to quantify the incremental benefit of B independently.
## How was this patch tested?
- `./dev/format-scala-code.sh`
- `./dev/format-cpp-code.sh` equivalent with clang-format-15 on all modified
C++ files
- `./dev/gen-all-config-docs.sh`
- Spark 3.5 / Scala 2.12 Delta test compilation
- Spark 4.0 / Scala 2.13 Delta test compilation
- Spotless and clang-format-15 dry-run checks
- Added JVM descriptor-handoff, native stored-range validation, split
conversion, runtime-config, and cross-Delta-version tests
A full local native build is blocked by an unrelated Homebrew glog header
mismatch on this macOS host; Linux native CI is authoritative for the C++ build
and tests.
## Generative AI tooling
Generated-by: IBM BOB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
