Peter Toth created SPARK-59905:
----------------------------------
Summary: Storage-partitioned join sorts partitions the wrong way
for an ORDER BY over a join-key expression
Key: SPARK-59905
URL: https://issues.apache.org/jira/browse/SPARK-59905
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
With spark.sql.sources.v2.bucketing.shuffle.enabled and
spark.sql.sources.v2.bucketing.sorting.enabled on, this query returns its rows
in reverse order. {{ident}} is identity-partitioned on {{id}} and holds 94 to
100, and {{plain(b)}} holds 0 to 6:
{code:sql}
SELECT p.b FROM ident i JOIN plain p ON i.id = 100 - p.b ORDER BY 100 - p.b
{code}
It returns 0, 1, ..., 6 instead of 6, 5, ..., 0.
The join shuffles {{plain}} onto {{ident}}'s layout, so
{{KeyedShuffleSpec.createPartitioning}} reports the bare join key {{100 - b}}
as {{plain}}'s partition expression. Its key rows then already hold {{100 -
b}}. When {{EnsureRequirements}} checks the keyed child against the
{{OrderedDistribution}}, it binds the required ordering to
{{expressions.flatMap(_.references)}}, which is {{b}}. So it computes {{100 -
(100 - b)}} on the key rows and orders the partitions the wrong way. The global
sort is removed all the same.
Reported by dongjoon-hyun in the review of
https://github.com/apache/spark/pull/59165.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
