Yahya Kisana created SPARK-60411:
------------------------------------
Summary: Share the ProjectionOverSchema rebinding of DSv2
expressions to a pruned scan output
Key: SPARK-60411
URL: https://issues.apache.org/jira/browse/SPARK-60411
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yahya Kisana
Three places rebind an expression to a scan's pruned output with the same
pattern: project it with {{ProjectionOverSchema}}, catch
{{SparkIllegalArgumentException}} with condition {{FIELD_NOT_FOUND}} for a
nested field that pruning removed, and keep the result only if its references
are still in the output:
* {{DataSourceV2ScanRelation.outputBoundPushedFilters}}
* the inferred-filter remap in {{V2ScanRelationPushDown}}
* {{DataSourceV2ScanExecBase.bindToOutput}} (SPARK-60011), which also drops a
result whose type pruning narrowed and catches any {{NonFatal}} error
The copies already differ in shape, and the pattern broke once before:
SPARK-56385 failed with {{FIELD_NOT_FOUND}} after nested schema pruning until
its follow-up added the catch. A method on {{ProjectionOverSchema}}, such as
{{projectOrNone(expr): Option[Expression]}}, would keep the projection and the
{{FIELD_NOT_FOUND}} handling in one place, while each caller keeps its own
extra checks.
Suggested in the review of SPARK-60011
(https://github.com/apache/spark/pull/59387).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
