felipepessoto opened a new pull request, #13512:
URL: https://github.com/apache/gluten/pull/13512
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after
updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the
corresponding issue
when the PR is merged.
-->
Relates to #12538.
Enable actual native Delta writing of `TIMESTAMP_NTZ`, rather than treating
a Spark fallback as native support.
- Preserve the NTZ logical-type metadata across Spark's Arrow schema
boundary. Reconstruct the native file schema using Arrow timestamp timezone
semantics, including map keys whose metadata is lost during Arrow C++ import.
- Write NTZ as Parquet `INT64` / `TIMESTAMP(MICROS, isAdjustedToUTC=false)`
while retaining `isAdjustedToUTC=true` for ordinary timestamps in mixed schemas.
- Prepare only affected writer columns containing null structs using
existing vector copy/null APIs. This prevents Arrow conversion of inaccessible
child payloads without mutating input vectors, changing visible values/types,
or suppressing reachable conversion errors.
- Generate native partition IDs from exact timestamp microseconds for both
NTZ and ordinary TIMESTAMP keys. Original vectors still supply file payloads
and partition paths.
- Allow direct field-access chains and their null predicates through mixed
NTZ structs so their statistics preprojection can execute natively.
- Remove the blanket Delta 4 NTZ rejection while retaining explicit
NTZ-validation opt-out and the existing option, field-ID,
partition-materialization and format gates.
The writer preparation adds one logical column copy plus null/offset
traversal for each affected top-level column in a batch. Unaffected columns and
batches reuse their existing vectors. This remains native writing; it does not
introduce a row-based Spark fallback.
**Scope:** no Velox pin or upstream source changes. The statistics fallback
work in #13510 remains separate. The literal-preservation work in #13041
remains separate: `Validators.scala` is the only file overlap, in different
eligibility logic; no literal serialization changes are copied or required.
This does not fix reading old DATE files after DATE-to-NTZ widening.
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
Rebuilt the unchanged pinned Velox
`bd5d1828bc100d7c3d83c0fb77ed15a1c2e8b8c4` and Gluten native libraries in an
isolated LF Linux checkout, using the same CentOS 9 / JDK 17 / GCC toolset 13
dependency environment for compilation and runtime.
| Validation | Result |
|---|---|
| Native writer utility GTests, compiled with the native build's CMake
compiler flags and linked against the rebuilt libraries | 7 passed |
| Spark 4.1.1 / Delta 4.4.1: NTZ writer and existing partition suites | 20
passed |
| Spark 4.1.1 / Delta 4.4.1: writer gate and existing Delta 4 writer suites
| 30 passed |
| Spark 3.5.5 / Delta 3.3.2: NTZ writer and existing partition suites | 20
passed |
| Spark 3.5.5 / Delta 3.3.2: existing native NTZ statistics suite | 1 passed
|
The tests cover row-backed and native-scan inputs, create/append, structs,
arrays, map keys/values, mixed NTZ/LTZ schemas, null parents, microseconds,
pre-epoch values and three session timezones. Partition coverage includes
integer, NTZ and LTZ keys, repeated/null/pre-epoch keys across small batches,
and both primitive and nested payloads.
Files must identify `parquet-cpp-velox` as their writer. Every timestamp
leaf's physical/logical type is checked. Data, inferred schemas, relative
partition directories, partition values and per-file statistics are compared
with independent Spark/Delta writes while Gluten is disabled for the oracle.
The ancient LTZ fixture uses an explicit corrected-calendar reader because
native files do not carry Spark writer-version/rebase metadata.
Native tests verify that original vectors and visible values/types survive
preparation, unaffected columns/batches are reused, a non-null outer struct can
contain a nullable inner struct, and reachable timestamp conversion errors
still throw even when another struct row is null.
As a negative control, the unchanged direct Arrow-export probe still
reproduces the pinned exporter's null-parent failure after rebuilding stock
`Bridge.cpp`. The writer-boundary tests pass with that same unpatched
dependency.
Also passed:
- `bash dev/format-cpp-code.sh` with clang-format 15.0.7
- `bash dev/format-scala-code.sh`
- `python3 .github/workflows/util/check.py header main --fix`
- `git diff --check`
These are targeted native/Delta regressions, not the full Delta suite,
GPU/cloud-connector validation, or the historical Spark 4.1.0 / Delta 4.2.0
DATE-to-NTZ ALTER scenario.
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: GitHub Copilot (GPT-6 Astra, gpt-6-astra)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
