felipepessoto opened a new pull request, #12626:
URL: https://github.com/apache/gluten/pull/12626
## What changes are proposed in this pull request?
Gluten's file-source scan transformers push **every** filter conjunct into
the native scan
via `PushDownFilterToScan` and evaluate them as exact row-level filters, so
the paired
`FilterExecTransformer` becomes a no-op (`FilterExecTransformerBase.isNoop`)
and no separate
post-scan `Filter` is needed. This is exactly the state Spark marks with a
leading `*` in a
scan's `PushedFilters` metadata — `RowDataSourceScanExec`'s convention for a
filter the source
fully handles itself.
Gluten inherited `FileSourceScanLike`'s rendering, which leaves
`PushedFilters` unmarked, so its
plan output under-reported what the native scan actually does. This PR marks
each pushed-filter
entry with `*` in the rendered plan string of
`FileSourceScanExecTransformerBase`. Because
`metadata` is a `lazy val` and cannot be `super`-overridden, the marking is
applied to the
rendered node string in both paths that print it:
- `simpleString` (`executedPlan.toString`), and
- `verboseStringWithOperatorId` (`FormattedMode` / plan-stability golden
files).
Only `FileSourceScanExecTransformer` nodes are marked. AQE `FormattedMode`
also prints an
*Initial Plan* whose scans are vanilla `FileSourceScanExec` (`Scan
parquet`); those are left
unmarked, which matches runtime output and the semantics of `*` (the vanilla
Initial-Plan scan
still has a real `Filter` above it).
Golden plans are updated accordingly (TPC-H approved-plan, TPC-DS
plan-stability, and
gluten-tpch-plan-stability across the supported Spark versions): every
offloaded
`FileSourceScanExecTransformer` `PushedFilters` entry now carries `*`, while
vanilla Initial-Plan
scan entries stay unmarked. The goldens contain no DataSource V2 `BatchScan`
`PushedFilters`, so
no V2 golden is affected.
## How was this patch tested?
- `gluten-substrait` compiles against `main` (`spark-3.5` / `scala-2.12`).
- Golden plans regenerated deterministically and cross-checked: **0**
vanilla `Scan parquet` nodes
marked and **0** `FileSourceScanExecTransformer` nodes left unmarked
across all 1672 golden
files. The golden diff is purely the `*` toggling (no unrelated plan
drift).
- The updated `PushedFilters` rendering is exercised by the existing
`VeloxTPCHSuite` golden-file
checks and the TPC-DS / TPC-H plan-stability suites.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
