[
https://issues.apache.org/jira/browse/SPARK-60011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125445#comment-18125445
]
Peter Toth commented on SPARK-60011:
------------------------------------
Sure, please go ahead [~ykisana]. Thanks!
> Keep a DSv2 nested-field ordering after schema pruning
> ------------------------------------------------------
>
> Key: SPARK-60011
> URL: https://issues.apache.org/jira/browse/SPARK-60011
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Yahya Kisana
> Priority: Minor
>
> [SPARK-59948|https://issues.apache.org/jira/projects/SPARK/issues/SPARK-59948]
> strips the Alias around a nested sort key, so a DSv2 scan's reported
> ordering on s.x can satisfy a required ordering on s.x. This only works while
> the scan keeps the whole struct. When nested schema pruning narrows the
> struct, Spark still adds a redundant
> SortExec.
> With s struct<x: bigint, y: bigint> and a reported ordering on s.x:
> |query|scan output s|SortExec|
> |SELECT * FROM t SORT BY s.x|struct<x, y>|no|
> |SELECT id, s FROM t SORT BY s.x|struct<x, y>|no|
> |SELECT id, s.x FROM t SORT BY s.x|struct<x>|yes|
>
> *Cause:* V2ScanPartitioningAndOrdering resolves the ordering against the full
> DataSourceV2Relation, so its s attribute keeps the unpruned type. The
> required ordering uses s from the pruned scan output. AttributeReference
> equality includes the data type, so the two GetStructFields never match.
> Rebinding s to the scan output by exprId is not enough on its own: the
> GetStructField ordinal also comes from the full struct and must be recomputed
> against the pruned type.
> Query results are correct. The cost is an unnecessary sort.
> Found by [~petertoth] while reviewing
> https://github.com/apache/spark/pull/59199.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
