felipepessoto opened a new pull request, #13502:
URL: https://github.com/apache/gluten/pull/13502
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after
updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the
corresponding issue
when the PR is merged.
-->
Fixes #13500.
Add Velox native-write support for Delta's [Materialize Partition
Columns](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#materialize-partition-columns)
writer feature in the Spark 4.x / Delta 4 integration. **This PR remains a
draft pending native integration validation.**
Delta already resolves feature/property precedence and includes partition
columns in the requested file schema. The native splitter previously discarded
those columns regardless of that decision.
- Forward the existing `reservePartitionColumns` argument through the
Scala/Java/JNI boundary. Retain partition columns in the native output batch
after the regular data columns, matching Delta's file schema even when input
columns or partition keys have a different order.
- Give statistics trackers a separate data-only batch, retaining their
existing non-partition schema and excluding partition columns from Delta file
statistics. Preserve batch ownership and per-file slicing.
- Replace the blanket materialization fallback with targeted partition-type
restrictions. Keep column mapping, Iceberg compatibility, and the other
existing safety fallbacks. FLOAT/DOUBLE partition columns fall back explicitly;
materialized TIMESTAMP partition columns remain on Delta's writer because the
native splitter cannot represent sub-millisecond values.
- Cover feature/property precedence, native Parquet schemas and values,
null/empty-string and multiple partition keys, file splitting and statistics,
mixed old/new files, and UPDATE/DELETE parity with vanilla Delta. Let the
existing Delta 4 native-write suite exercise Delta's materialization default
and update the support documentation.
This does not enable full UniForm support or change the Delta 3.3 writer's
materialization behavior.
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
- Scala/Java main and test compilation completed for `backends-velox` and
`gluten-ut/spark41` with Spark 4.1, Scala 2.13, and JDK 17.
- `org.apache.gluten.delta.GlutenDeltaNativeWriteGateSuite`: **19 tests
passed**.
- `dev/format-scala-code.sh` and its `check` mode passed with an
affected-file selector; `dev/format-cpp-code.sh` ran with clang-format 15.0.7;
the repository license-header check and `git diff --check` passed.
- **Native end-to-end execution is not yet validated.** The native build
attempt was blocked locally by `cmake: command not found`. The JNI library must
be rebuilt before running `VeloxDelta4WriteSuite`, `DeltaNativeWriteSuite`, and
`GlutenDeltaNativePartitionWriteSuite`; the added integration cases have been
compiled but not executed.
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: GitHub Copilot 1.0.102-0.unstable.r40141230562.g130d689 (GPT-6
Astra / gpt-6-astra)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
