szehon-ho opened a new pull request, #57340:
URL: https://github.com/apache/spark/pull/57340
### What changes were proposed in this pull request?
This PR adds an optional `fullyPushedFilters` hint to the `FileIndex` trait
and
teaches `FileSourceStrategy` to use it when deciding which filters can be
removed
from the row-level `FilterExec` placed above the scan.
- `FileIndex.fullyPushedFilters: Option[Seq[Expression]]` defaults to `None`.
- In `FileSourceStrategy`, the set of post-scan (after-scan) filters is now
computed as:
- When the `FileIndex` returns `Some(fullyPushedFilters)`, those filters
are the
only basis for removing predicates from the `FilterExec` above the scan
(`filterSet -- fullyPushedFilters`).
- When it returns `None`, planning keeps the existing default behavior
(`filterSet -- partitionKeyFilters.filter(_.references.nonEmpty)`).
This lets a `FileIndex` implementation that guarantees certain predicates
for all
returned rows (e.g. an index that fully prunes on non-partition columns)
tell the
planner exactly which filters are safe to drop, instead of relying only on
the
default partition-key heuristic.
### Why are the changes needed?
Today `FileSourceStrategy` assumes only partition-key filters are fully
satisfied by
file listing, so any other predicate a `FileIndex` may fully guarantee must
still be
re-evaluated in a `FilterExec`. Custom `FileIndex` implementations have no
way to
communicate stronger pushdown guarantees to the physical plan. The new hint
provides
an opt-in mechanism, with no behavior change for existing indexes (default
`None`).
### Does this PR introduce _any_ user-facing change?
No. The new trait method has a default implementation returning `None`, which
preserves the existing planning behavior for all built-in `FileIndex`
implementations.
### How was this patch tested?
Added a unit test in `FileSourceStrategySuite`
("FileIndex fullyPushedFilters controls final FilterExec removal") that
exercises a
test `FileIndex` returning `None`, `Some(Nil)`, and `Some(<data filter>)`,
and asserts
the resulting `FilterExec` predicates. Ran:
build/sbt 'sql/testOnly *FileSourceStrategySuite -- -z
"fullyPushedFilters"'
All tests passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
