Weixin-Xu opened a new pull request, #13380:
URL: https://github.com/apache/gluten/pull/13380
## What changes are proposed in this pull request?
This PR fixes expression translation and generator rewrites in the Bolt
backend to preserve Spark behavior.
- Preserve exact `from_json` schema field names recursively, including
nested structs, arrays, and maps.
- Forward Spark's Base64 chunking configuration to Bolt and fall back when
an expression's bound option differs from the native query configuration.
- Preserve the resolved expression time zone and null-format behavior for
DATE unix-timestamp expressions.
- Translate empty maps as literals, align Bloom filter capacity with the
native aggregate, and allow `bitmap_construct_agg` through native aggregate
validation.
- Apply generator project rewrites after transformation, preserving
aliases, metadata, output order, and input/filter bindings. Keep
nondeterministic and potentially throwing expressions in their original
evaluation positions.
- Correct `json_tuple` post-projection handling and assign native stage
IDs after stage collapse.
- Replace the missing explode test resource with a temporary fixture.
Add an opt-in backend hook for Spark version StaticInvoke expressions. Its
default returns `None`, preserving the existing fallback behavior for other
backends; Bolt opts into the existing Spark version
literal.
## How was this patch tested?
These changes were validated as part of the integrated Bolt CI candidate
using Bolt revision `071e54764f887dfb4337260e440f2f12f6ab82f5` with the
recorded engine fixes applied.
Added regression coverage includes:
- JSON field-name casing, nested schemas, and unchanged generic schema
serialization.
- Base64 defaults, explicit settings, and expression-bound options after
session configuration changes.
- DATE time-zone handling and preservation of existing TIMESTAMP and
STRING overloads.
- Spark version translation and fallback for unrelated signatures or
backends using the default hook.
- Generator aliases, bindings, output order, nondeterministic evaluation,
and errors in discarded rows.
- `json_tuple` projection handling and native stage-ID assignment.
The integrated candidate completed the reference Spark test coverage.
After rebasing this topic onto Apache Gluten main, `range-diff` and stable
patch IDs confirmed that the patch was preserved. Applying the original patch
to the new base reproduced the rebased tree exactly, and
`git diff --check` passed.
The standalone rebased branch has not been rebuilt or runtime-tested
independently.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
