[
https://issues.apache.org/jira/browse/SPARK-60944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated SPARK-60944:
------------------------------
Description:
DSv2 sources can report inferred Catalyst filters for statistics estimation
after SPARK-58892, including a mode that keeps them on logical Filter nodes.
These predicates can remain in executable filters even when the source already
enforces them.
Add an internal table capability exposing fullyPushedFilterAttributes() for
both static and Catalyst runtime filtering. For scans reporting inferred
filters, DataSourceV2Strategy can omit eligible static filter conjuncts whose
references all belong to that attribute set, while preserving logical filters
for statistics. Sources must fully enforce query filters over declared columns.
Keep Boolean projections executable, including aliases created by
MergeSubplans, and retain static filters for scans without inference. Deprecate
the scan-level runtime declaration with migration documentation and a
compatibility fallback.
Tracking the originally inferred expressions is fragile: BooleanSimplification
can factor them together with residual predicates; MergeSubplans can put branch
predicates into Boolean aliases and rebuild scans; and projection,
normalization, and schema pruning can change references. Attribute-based
checking avoids expression-provenance bookkeeping. Mixed-column expressions and
inferred predicates inside merge aliases can still be evaluated.
Original PR and discussion: https://github.com/apache/spark/pull/58145
Implementation: https://github.com/apache/spark/pull/60147
was:
DSv2 sources can report inferred Catalyst filters for statistics estimation
after SPARK-58892, including a mode that keeps them on logical Filter nodes.
These predicates can remain in executable filters even when the source already
guarantees that every returned row satisfies them.
Add an internal table capability that proves whether a final Catalyst predicate
is SQL true for every row returned by a supplied built Scan.
DataSourceV2Strategy can omit proven conjuncts from physical filters and
Boolean projection aliases while leaving logical statistics unchanged. Keep
removal conservative when the source cannot prove the final expression true.
Tracking the originally inferred expressions is insufficient:
BooleanSimplification can factor them together with residual predicates;
MergeSubplans can put branch predicates into Boolean aliases and rebuild scans;
and projection, normalization, and schema pruning can change attribute
references. A guarantee from an old scan is not necessarily valid for its
replacement. The callback checks the final expression against the actual scan
instead of reconstructing its provenance.
The table capability also hosts fullyPushedFilterAttributes(), shared with
Catalyst runtime filtering. Its attribute declaration applies to predicates
actually sent to the runtime-filtering scan; static predicates require the
separate per-expression proof.
Original PR and discussion: https://github.com/apache/spark/pull/58145
Summary: Skip DSv2 filters on fully pushed attributes (was: Skip
physical evaluation of guaranteed DSv2 filters)
> Skip DSv2 filters on fully pushed attributes
> --------------------------------------------
>
> Key: SPARK-60944
> URL: https://issues.apache.org/jira/browse/SPARK-60944
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> DSv2 sources can report inferred Catalyst filters for statistics estimation
> after SPARK-58892, including a mode that keeps them on logical Filter nodes.
> These predicates can remain in executable filters even when the source
> already enforces them.
> Add an internal table capability exposing fullyPushedFilterAttributes() for
> both static and Catalyst runtime filtering. For scans reporting inferred
> filters, DataSourceV2Strategy can omit eligible static filter conjuncts whose
> references all belong to that attribute set, while preserving logical filters
> for statistics. Sources must fully enforce query filters over declared
> columns. Keep Boolean projections executable, including aliases created by
> MergeSubplans, and retain static filters for scans without inference.
> Deprecate the scan-level runtime declaration with migration documentation and
> a compatibility fallback.
> Tracking the originally inferred expressions is fragile:
> BooleanSimplification can factor them together with residual predicates;
> MergeSubplans can put branch predicates into Boolean aliases and rebuild
> scans; and projection, normalization, and schema pruning can change
> references. Attribute-based checking avoids expression-provenance
> bookkeeping. Mixed-column expressions and inferred predicates inside merge
> aliases can still be evaluated.
> Original PR and discussion: https://github.com/apache/spark/pull/58145
> Implementation: https://github.com/apache/spark/pull/60147
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
