[
https://issues.apache.org/jira/browse/SPARK-59981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59981:
-----------------------------------
Labels: pull-request-available (was: )
> A V2 scan that reports an empty ordering loses the ordering derived from its
> partition keys
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-59981
> URL: https://issues.apache.org/jira/browse/SPARK-59981
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> With {{spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled}} on (the
> default since SPARK-59396), a V2 scan that reports a
> {{KeyGroupedPartitioning}} gets an output ordering over its partition key
> expressions (SPARK-56241). That works when the scan does not implement
> {{SupportsReportOrdering}}, or when Spark ignores the ordering it reports
> (SPARK-59721). It does not work when the scan implements
> {{SupportsReportOrdering}} and reports an empty ordering:
> * {{V2ScanPartitioningAndOrdering}} stores the empty report as {{Some(Nil)}}.
> * {{DataSourceV2ScanExecBase.outputOrdering}} takes its {{case (Some(o), p)}}
> branch before the derivation, so the scan reports no ordering at all.
> Measured with a scan that reports {{identity(id)}}, joined on {{id}} with
> another {{identity(id)}} table: the sort-merge join keeps a sort above the
> scan. When the same scan reports an ordering that Spark ignores, the ordering
> is derived and the sort goes away. The results are correct either way.
> The conf's doc covers this case: "a V2 data source that reports a
> KeyedPartitioning but does not report explicit ordering via
> SupportsReportOrdering".
> Found in the review of https://github.com/apache/spark/pull/58979.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
