Anil K created SPARK-62150:
------------------------------
Summary: A partition transform of a collated string column lets a
window or aggregate skip its shuffle and split values equal under the collation
Key: SPARK-62150
URL: https://issues.apache.org/jira/browse/SPARK-62150
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Anil K
A V2 scan partitioned by a transform other than {{identity}} of a string column
with a non-binary collation, e.g. {{bucket(n, s)}} over {{s STRING COLLATE
UTF8_LCASE}}, reports a {{KeyedPartitioning}} that satisfies a
{{ClusteredDistribution}} on {{s}}. A window or an aggregate over {{s}} then
skips its shuffle and runs on the scan's partitions. That is correct only if
values equal under the collation land in one partition. Spark merges partitions
whose keys compare equal, and compares string keys with the collation, so
{{identity(s)}} is fine. But a transform can map two values equal under the
collation to keys that are not equal:
* {{bucket}} over a hash of the raw bytes puts {{'A'}} and {{'a'}} into
different buckets;
* a prefix, such as a parameterless {{prefix1(s)}} or {{truncate(s, 1)}}, maps
{{U+0130}} ({{İ}}) and {{i U+0307}}, which are equal under {{UTF8_LCASE}}, to
{{'İ'}} and {{'i'}}, which are not.
Measured on master (98bbfe57a20) with the test catalog,
{{spark.sql.sources.v2.bucketing.enabled=true}}, one split per row, against the
same query with bucketing disabled:
||transform||rows||{{count(*) OVER (PARTITION BY s)}}||{{GROUP BY
s}}||shuffles||
|{{bucket(5, s)}}|{{'A'}}, {{'a'}}|1, 1 (expected 2, 2)|2 groups of 1 (expected
1 group of 2)|0|
|{{bucket(5, s)}}|{{U+0130}}, {{i U+0307}}|1, 1|2 groups of 1|0|
|{{prefix1(s)}}|{{U+0130}}, {{i U+0307}}|1, 1|2 groups of 1|0|
|{{identity(s)}}|either|2, 2|1 group of 2|0|
An equi-join on {{s}} is correct: it is planned over collation keys, so both
sides shuffle.
SPARK-50593 (https://github.com/apache/spark/pull/55885) lets a transform with
literal parameters, such as {{truncate(s, w)}}, report its partitioning, so
{{truncate(s, 1)}} gives the same wrong rows as {{prefix1(s)}} above, where it
shuffled before.
A fix could make a transform other than {{identity}} over a column whose type
is not binary stable ({{UnsafeRowUtils.isBinaryStable}}) not satisfy a
clustering on that column, e.g. in {{KeyedPartitioning.supportsExpressions}},
so such a scan shuffles. {{identity}} can keep its partitioning, since its keys
are compared with the collation. A connector that knows its transform respects
the collation has no way to say so today, so this costs those transforms their
shuffle-free plans too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]