[
https://issues.apache.org/jira/browse/SPARK-60944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated SPARK-60944:
------------------------------
Description:
DSv2 sources can report inferred Catalyst filters for statistics estimation
after SPARK-58892, including a mode that keeps them on logical Filter nodes.
These predicates can remain in executable filters even when the source already
enforces them.
Add an internal table capability exposing fullyPushedFilterAttributes() for
static filter removal. For scans reporting inferred filters,
DataSourceV2Strategy can omit eligible static filter conjuncts whose accessed
fields are all covered by that declaration, while preserving logical filters
for statistics. Sources must fully enforce query filters over declared columns.
Keep Boolean projections executable, including aliases created by
MergeSubplans, and retain static filters for scans without inference. Support
top-level columns and exact nested struct paths, preserving sibling filters and
resolving fields against the pruned scan schema. Keep the existing runtime scan
declaration separate and unchanged, including its top-level restriction.
Tracking the originally inferred expressions is fragile: BooleanSimplification
can factor them together with residual predicates; MergeSubplans can put branch
predicates into Boolean aliases and rebuild scans; and projection,
normalization, and schema pruning can change references. Attribute-based
checking avoids expression-provenance bookkeeping. Mixed-column expressions and
inferred predicates inside merge aliases can still be evaluated.
Original PR and discussion: https://github.com/apache/spark/pull/58145
Implementation: https://github.com/apache/spark/pull/60147
was:
DSv2 sources can report inferred Catalyst filters for statistics estimation
after SPARK-58892, including a mode that keeps them on logical Filter nodes.
These predicates can remain in executable filters even when the source already
enforces them.
Add an internal table capability exposing fullyPushedFilterAttributes() for
both static and Catalyst runtime filtering. For scans reporting inferred
filters, DataSourceV2Strategy can omit eligible static filter conjuncts whose
references all belong to that attribute set, while preserving logical filters
for statistics. Sources must fully enforce query filters over declared columns.
Keep Boolean projections executable, including aliases created by
MergeSubplans, and retain static filters for scans without inference. Deprecate
the scan-level runtime declaration with migration documentation and a
compatibility fallback.
Tracking the originally inferred expressions is fragile: BooleanSimplification
can factor them together with residual predicates; MergeSubplans can put branch
predicates into Boolean aliases and rebuild scans; and projection,
normalization, and schema pruning can change references. Attribute-based
checking avoids expression-provenance bookkeeping. Mixed-column expressions and
inferred predicates inside merge aliases can still be evaluated.
Original PR and discussion: https://github.com/apache/spark/pull/58145
Implementation: https://github.com/apache/spark/pull/60147
> Skip DSv2 filters on fully pushed attributes
> --------------------------------------------
>
> Key: SPARK-60944
> URL: https://issues.apache.org/jira/browse/SPARK-60944
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> DSv2 sources can report inferred Catalyst filters for statistics estimation
> after SPARK-58892, including a mode that keeps them on logical Filter nodes.
> These predicates can remain in executable filters even when the source
> already enforces them.
> Add an internal table capability exposing fullyPushedFilterAttributes() for
> static filter removal. For scans reporting inferred filters,
> DataSourceV2Strategy can omit eligible static filter conjuncts whose accessed
> fields are all covered by that declaration, while preserving logical filters
> for statistics. Sources must fully enforce query filters over declared
> columns. Keep Boolean projections executable, including aliases created by
> MergeSubplans, and retain static filters for scans without inference. Support
> top-level columns and exact nested struct paths, preserving sibling filters
> and resolving fields against the pruned scan schema. Keep the existing
> runtime scan declaration separate and unchanged, including its top-level
> restriction.
> Tracking the originally inferred expressions is fragile:
> BooleanSimplification can factor them together with residual predicates;
> MergeSubplans can put branch predicates into Boolean aliases and rebuild
> scans; and projection, normalization, and schema pruning can change
> references. Attribute-based checking avoids expression-provenance
> bookkeeping. Mixed-column expressions and inferred predicates inside merge
> aliases can still be evaluated.
> Original PR and discussion: https://github.com/apache/spark/pull/58145
> Implementation: https://github.com/apache/spark/pull/60147
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
