[
https://issues.apache.org/jira/browse/SPARK-59905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-59905:
-------------------------------
Description:
With spark.sql.sources.v2.bucketing.shuffle.enabled and
spark.sql.sources.v2.bucketing.sorting.enabled on, this query returns its rows
in reverse order. {{ident}} is identity-partitioned on {{id}} and holds 94 to
100, and {{plain(b)}} holds 0 to 6:
{code:sql}
SELECT p.b FROM ident i JOIN plain p ON i.id = 100 - p.b ORDER BY 100 - p.b
{code}
It returns 0, 1, ..., 6 instead of 6, 5, ..., 0.
The join shuffles {{plain}} onto {{ident}}'s layout, so
{{KeyedShuffleSpec.createPartitioning}} reports the bare join key {{100 - b}}
as {{plain}}'s partition expression. Its key rows then already hold {{100 -
b}}. When {{EnsureRequirements}} checks the keyed child against the
{{OrderedDistribution}}, it binds the required ordering to
{{expressions.flatMap(_.references)}}, which is {{b}}. So it computes {{100 -
(100 - b)}} on the key rows and orders the partitions the wrong way. The global
sort is removed all the same.
The same binding also fails planning for a join key over two columns. With
{{ident}} holding 0 to 7 and {{plain2(b, c)}} holding (0, 1), (0, 2), (1, 1)
and (1, 2):
{code:sql}
SELECT * FROM ident i JOIN plain2 p ON i.id = p.b + p.c ORDER BY p.b + p.c
{code}
It fails with {{ArrayIndexOutOfBoundsException: Index 1 out of bounds for
length 1}}. The partition expression is {{b + c}}, so binding the ordering to
its references gives two attributes for a key row with one value.
Reported by dongjoon-hyun in the review of
https://github.com/apache/spark/pull/59165.
was:
With spark.sql.sources.v2.bucketing.shuffle.enabled and
spark.sql.sources.v2.bucketing.sorting.enabled on, this query returns its rows
in reverse order. {{ident}} is identity-partitioned on {{id}} and holds 94 to
100, and {{plain(b)}} holds 0 to 6:
{code:sql}
SELECT p.b FROM ident i JOIN plain p ON i.id = 100 - p.b ORDER BY 100 - p.b
{code}
It returns 0, 1, ..., 6 instead of 6, 5, ..., 0.
The join shuffles {{plain}} onto {{ident}}'s layout, so
{{KeyedShuffleSpec.createPartitioning}} reports the bare join key {{100 - b}}
as {{plain}}'s partition expression. Its key rows then already hold {{100 -
b}}. When {{EnsureRequirements}} checks the keyed child against the
{{OrderedDistribution}}, it binds the required ordering to
{{expressions.flatMap(_.references)}}, which is {{b}}. So it computes {{100 -
(100 - b)}} on the key rows and orders the partitions the wrong way. The global
sort is removed all the same.
Reported by dongjoon-hyun in the review of
https://github.com/apache/spark/pull/59165.
> Storage-partitioned join sorts partitions the wrong way for an ORDER BY over
> a join-key expression
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-59905
> URL: https://issues.apache.org/jira/browse/SPARK-59905
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Priority: Major
>
> With spark.sql.sources.v2.bucketing.shuffle.enabled and
> spark.sql.sources.v2.bucketing.sorting.enabled on, this query returns its
> rows in reverse order. {{ident}} is identity-partitioned on {{id}} and holds
> 94 to 100, and {{plain(b)}} holds 0 to 6:
> {code:sql}
> SELECT p.b FROM ident i JOIN plain p ON i.id = 100 - p.b ORDER BY 100 - p.b
> {code}
> It returns 0, 1, ..., 6 instead of 6, 5, ..., 0.
> The join shuffles {{plain}} onto {{ident}}'s layout, so
> {{KeyedShuffleSpec.createPartitioning}} reports the bare join key {{100 - b}}
> as {{plain}}'s partition expression. Its key rows then already hold {{100 -
> b}}. When {{EnsureRequirements}} checks the keyed child against the
> {{OrderedDistribution}}, it binds the required ordering to
> {{expressions.flatMap(_.references)}}, which is {{b}}. So it computes {{100 -
> (100 - b)}} on the key rows and orders the partitions the wrong way. The
> global sort is removed all the same.
> The same binding also fails planning for a join key over two columns. With
> {{ident}} holding 0 to 7 and {{plain2(b, c)}} holding (0, 1), (0, 2), (1, 1)
> and (1, 2):
> {code:sql}
> SELECT * FROM ident i JOIN plain2 p ON i.id = p.b + p.c ORDER BY p.b + p.c
> {code}
> It fails with {{ArrayIndexOutOfBoundsException: Index 1 out of bounds for
> length 1}}. The partition expression is {{b + c}}, so binding the ordering to
> its references gives two attributes for a key row with one value.
> Reported by dongjoon-hyun in the review of
> https://github.com/apache/spark/pull/59165.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
