felipepessoto opened a new pull request, #12539:
URL: https://github.com/apache/gluten/pull/12539
## What changes are proposed in this pull request?
Fixes #12538.
`GlutenDeltaJobStatsTracker` collects Delta write statistics by building a
local Velox aggregation plan and assuming it is always offloaded to a
`WholeStageTransformer`. When the statistics plan references a type or
expression Velox cannot offload — for example an aggregate over a
`TIMESTAMP_NTZ` column — the offload rules reject it and leave a vanilla
`ProjectExec`, so the unconditional cast threw:
```
java.lang.ClassCastException: class
org.apache.spark.sql.execution.ProjectExec cannot be cast to
class org.apache.gluten.execution.WholeStageTransformer
at
...GlutenDeltaJobStatsTracker$GlutenDeltaTaskStatsTracker.<init>(GlutenDeltaJobStatsTracker.scala:174)
at
...GlutenDeltaJobStatsTracker.newTaskInstance(GlutenDeltaJobStatsTracker.scala:80)
```
failing the whole write.
This PR makes the Delta stats path fall back to row-based statistics
collection when the plan cannot be fully offloaded, as suggested by @rui-mo and
@zhztheplayer in #12388:
- The decision is made on the executor in `newTaskInstance()`, where the
native plan is actually built.
- `tryBuildOffloadedStatsPlan` builds the aggregation plan, runs the offload
rules, and returns the offloaded `TransformSupport` **only when the whole plan
was offloaded** — every node except the `StatisticsInputNode` leaf is a
`TransformSupport`. A root `WholeStageTransformer` alone is not sufficient,
because `ColumnarCollapseTransformStages` can wrap an offloaded parent above a
vanilla child.
- Otherwise it returns `None` and statistics are collected the row-based way
through the existing `GlutenDeltaJobStatsFallbackTracker`. Native resources
(the single-thread executor and `NativePlanEvaluator`) are allocated only on
the accepted path.
Both the Delta 3.3 (`src-delta33`) and Delta 4.0+ (`src-delta40`) variants
are updated.
## How was this patch tested?
Added `DeltaTimestampNtzStatsWriteSuite` (both `src-delta33` and
`src-delta40`), which writes a Delta table with a top-level and a struct-nested
`TIMESTAMP_NTZ` column through the native write path with statistics collection
enabled. Without the fix the write throws the `ClassCastException` above; with
it the write succeeds, the data reads back correctly, and Delta
`minValues`/`maxValues` statistics are still produced for the `TIMESTAMP_NTZ`
column via the fallback.
Compiles for `-Pspark-4.1` (delta40) and `-Pspark-3.5` (delta33).
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
