felipepessoto opened a new pull request, #13510:
URL: https://github.com/apache/gluten/pull/13510
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid
duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR
title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after
updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the
corresponding issue
when the PR is merged.
-->
Fixes #12538.
The Delta statistics tracker assumes that its local statistics plan always
becomes a native whole stage. A rejected projection instead leaves a Spark
`ProjectExec`, causing the unchecked cast to fail. Checking only the root is
insufficient: disabling aggregation offload leaves a native projection above a
Spark aggregate, and the local evaluator then receives raw input batches where
it expects aggregate output.
For both Delta 3.3 and Delta 4, select the statistics implementation before
constructing the native task. Use native aggregation only when the complete
stage is transformable, with the expected statistics-input adapters. Otherwise
log the validation reasons and reuse `GlutenDeltaJobStatsFallbackTracker`,
which converts the incoming batches to rows for Delta's original tracker.
Native file writing and supported native statistics remain enabled.
Add a shared backend regression suite covering supported native statistics,
rejected projections, partially offloaded aggregation, and rejected NTZ
statistics. Extend the `gluten-ut` partitioned-write matrix to cover statistics
fallback across file rotation and batch splitting, and update the writer
documentation. Tests check actual writer provenance, file contents, row/null
counts, nested min/max, all-null columns, and native statistics execution
rather than only absence of exceptions.
The existing Delta 4 NTZ file-writer gate is unchanged; this does **not**
enable native NTZ file writing. No dependency pins, DATE-reader patches, CI
workflows, or upstream-test baseline entries are changed.
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
Built fresh native libraries from main commit
`cce7716962d3152b46fc22882bed4a3887a792ef` and its unchanged IBM Velox pin
`bd5d1828bc100d7c3d83c0fb77ed15a1c2e8b8c4`, before building the JVM modules.
Native build and runtime used the same `apache/gluten:centos-9-jdk17` builder
filesystem, GCC toolset 13 and JDK 17.0.20. No historical or reader-patched
native binaries were reused.
**Red/green:** with only the new tests overlaid on unmodified main,
`GlutenDeltaStatsFallbackSuite` ran four tests on Spark 4.1.1 / Delta 4.4.1:
the supported-native control passed, while the other three failed. Projection
and NTZ rejection reproduced `ProjectExec cannot be cast to
WholeStageTransformer`; aggregation-only rejection reproduced `(3 vs. 10) Value
stream batch has fewer columns than the declared output type`. All four pass
with this fix.
| Runtime | Suite / selection | Result |
|---|---|---|
| Spark 4.1.1 / Delta 4.4.1 / Scala 2.13.17 |
`GlutenDeltaStatsFallbackSuite` | 4/4 passed |
| Spark 4.1.1 / Delta 4.4.1 / Scala 2.13.17 |
`GlutenDeltaNativePartitionWriteSuite` | 5/5 passed |
| Spark 4.1.1 / Delta 4.4.1 / Scala 2.13.17 | Existing
`VeloxDelta4WriteSuite`: `TIMESTAMP_NTZ falls back, at any depth` | 1/1 passed |
| Spark 3.5.5 / Delta 3.3.2 / Scala 2.12.18 |
`GlutenDeltaStatsFallbackSuite` | 4/4 passed |
| Spark 3.5.5 / Delta 3.3.2 / Scala 2.12.18 | Existing
`GlutenDeltaStatsSuite` native NTZ statistics test | 1/1 passed |
| Spark 3.5.5 / Delta 3.3.2 / Scala 2.12.18 |
`GlutenDeltaNativePartitionWriteSuite` | 5/5 passed |
The JVM modules were built with `./build/mvn`, using the corresponding
Spark/Scala profiles plus `backends-velox,delta,spark-ut,fast-build`; Spark 4.1
also used `java-17` and `-Dmaven.compiler.release=17`. A clean reactor build
was used when switching to Spark 3.5. Actual Spark/Delta/Scala runtime versions
were checked in the JUnit classpaths. The tested Linux source tree matches the
committed tree (`21bfaef702a524bca532c37f969a6e922e86ddc0`).
Also passed `./dev/format-scala-code.sh`, the license-header check
(`check.py header main --fix`), and `git diff --cached --check`.
The historical Spark 4.1.0 / Delta 4.2.0 DATE-to-NTZ ALTER integration, the
full upstream Delta suite, and the other Spark 4.x profiles were **not** run.
The NTZ fallback regression exercises the statistics stream without bypassing
Delta 4's file-writer gate.
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html).
-->
Generated-by: GitHub Copilot (GPT-6 Astra, model ID: gpt-6-astra)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
