Peter Toth created SPARK-59981:
----------------------------------
Summary: A V2 scan that reports an empty ordering loses the
ordering derived from its partition keys
Key: SPARK-59981
URL: https://issues.apache.org/jira/browse/SPARK-59981
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
With {{spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled}} on (the
default since SPARK-59396), a V2 scan that reports a {{KeyGroupedPartitioning}}
gets an output ordering over its partition key expressions (SPARK-56241). That
works when the scan does not implement {{SupportsReportOrdering}}, or when
Spark ignores the ordering it reports (SPARK-59721). It does not work when the
scan implements {{SupportsReportOrdering}} and reports an empty ordering:
* {{V2ScanPartitioningAndOrdering}} stores the empty report as {{Some(Nil)}}.
* {{DataSourceV2ScanExecBase.outputOrdering}} takes its {{case (Some(o), p)}}
branch before the derivation, so the scan reports no ordering at all.
Measured with a scan that reports {{identity(id)}}, joined on {{id}} with
another {{identity(id)}} table: the sort-merge join keeps a sort above the
scan. When the same scan reports an ordering that Spark ignores, the ordering
is derived and the sort goes away. The results are correct either way.
The conf's doc covers this case: "a V2 data source that reports a
KeyedPartitioning but does not report explicit ordering via
SupportsReportOrdering".
Found in the review of https://github.com/apache/spark/pull/58979.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
