[
https://issues.apache.org/jira/browse/SPARK-59095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned SPARK-59095:
---------------------------------
Assignee: Szehon Ho
> Make runtime filter eligibility path-aware
> ------------------------------------------
>
> Key: SPARK-59095
> URL: https://issues.apache.org/jira/browse/SPARK-59095
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Szehon Ho
> Assignee: Szehon Ho
> Priority: Major
>
> Spark currently tracks runtime-filter eligibility in an AttributeSet, which
> reduces nested references to their root attributes. This is safe for ordinary
> filterAttributes() because Spark retains post-scan evaluation, but it is not
> safe for fullyPushedFilterAttributes(), where Spark removes the post-scan
> predicate. As a result, connectors cannot soundly declare one fully pushed
> nested path: declaring the root struct would also make predicates on sibling
> fields appear fully pushed. Make eligibility path-aware so Spark can
> distinguish nested paths and safely support fully pushed nested
> runtime-filter attributes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
