wirybeaver opened a new pull request, #12629: URL: https://github.com/apache/gluten/pull/12629
## What changes are proposed in this pull request? Fixes #12628. Add native Iceberg format-version 3 deletion-vector support to the Velox backend: - Serialize Puffin DV metadata through Substrait and convert it to Velox `FileContent::kDeletionVector`. - Replace blanket V3 scan fallback with per-delete validation while retaining fallback for equality deletes and incomplete metadata. - Offload Iceberg `WriteDeltaExec` for DELETE, UPDATE, and MERGE behind enhanced native writes. - Preserve Iceberg's existing `DeltaBatchWrite` coordinator for atomic commit, validation, conflict handling, and abort cleanup. - Seed native writers with existing DVs so repeated mutations union positions and replace the prior DV. - Preserve Spark 3.3 behavior and add Spark 3.4–4.1 compatibility shims, including Spark 4.1 MERGE summaries. - Add native, Scala, and enhanced end-to-end coverage, and run targeted native DV tests in enhanced CI. ## How was this patch tested? Passed: - `./dev/format-scala-code.sh` - `git diff --check` - Focused Spark 3.5 Iceberg tests: 2 JUnit and 8 Scala tests - Focused Spark 3.5 Velox backend tests: 11 tests - Spark 3.4 clean `test-compile` - Spark 4.0 / Scala 2.13 / Java 17 clean `test-compile` - Direct compatibility compilation of the V2 write shims against Spark 3.3, 3.4, 3.5, 4.0, and 4.1 - Direct compilation of the Spark 4.1 columnar V2 commit path and Spark 4.0 enhanced DV suite - Enhanced workflow YAML parsing Not completed locally: - Native C++ and enhanced end-to-end execution require a rebuilt enhanced Velox backend. - Spark 4.1 Maven tests are blocked by HTTP 403 responses for the Iceberg 1.11.0 test artifacts. - The license-header command is blocked because `dev/license-header.py` is absent from this checkout. - C++ formatting passed earlier with clang-format 15; it could not be rerun after the final JNI cleanup because clang-format is no longer available in the environment. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
