Yahya Kisana created SPARK-60011:
------------------------------------
Summary: Keep a DSv2 nested-field ordering after schema pruning
Key: SPARK-60011
URL: https://issues.apache.org/jira/browse/SPARK-60011
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Yahya Kisana
[SPARK-59948|https://issues.apache.org/jira/projects/SPARK/issues/SPARK-59948]
strips the Alias around a nested sort key, so a DSv2 scan's reported ordering
on s.x can satisfy a required ordering on s.x. This only works while the scan
keeps the whole struct. When nested schema pruning narrows the struct, Spark
still adds a redundant
SortExec.
With s struct<x: bigint, y: bigint> and a reported ordering on s.x:
|query|scan output s|SortExec|
|SELECT * FROM t SORT BY s.x|struct<x, y>|no|
|SELECT id, s FROM t SORT BY s.x|struct<x, y>|no|
|SELECT id, s.x FROM t SORT BY s.x|struct<x>|yes|
*Cause:* V2ScanPartitioningAndOrdering resolves the ordering against the full
DataSourceV2Relation, so its s attribute keeps the unpruned type. The required
ordering uses s from the pruned scan output. AttributeReference equality
includes the data type, so the two GetStructFields never match.
Rebinding s to the scan output by exprId is not enough on its own: the
GetStructField ordinal also comes from the full struct and must be recomputed
against the pruned type.
Query results are correct. The cost is an unnecessary sort.
Found by [~petertoth] while reviewing
https://github.com/apache/spark/pull/59199.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
