Anil K created SPARK-62150:
------------------------------

             Summary: A partition transform of a collated string column lets a 
window or aggregate skip its shuffle and split values equal under the collation
                 Key: SPARK-62150
                 URL: https://issues.apache.org/jira/browse/SPARK-62150
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Anil K


A V2 scan partitioned by a transform other than {{identity}} of a string column 
with a non-binary collation, e.g. {{bucket(n, s)}} over {{s STRING COLLATE 
UTF8_LCASE}}, reports a {{KeyedPartitioning}} that satisfies a 
{{ClusteredDistribution}} on {{s}}. A window or an aggregate over {{s}} then 
skips its shuffle and runs on the scan's partitions. That is correct only if 
values equal under the collation land in one partition. Spark merges partitions 
whose keys compare equal, and compares string keys with the collation, so 
{{identity(s)}} is fine. But a transform can map two values equal under the 
collation to keys that are not equal:
* {{bucket}} over a hash of the raw bytes puts {{'A'}} and {{'a'}} into 
different buckets;
* a prefix, such as a parameterless {{prefix1(s)}} or {{truncate(s, 1)}}, maps 
{{U+0130}} ({{İ}}) and {{i U+0307}}, which are equal under {{UTF8_LCASE}}, to 
{{'İ'}} and {{'i'}}, which are not.

Measured on master (98bbfe57a20) with the test catalog, 
{{spark.sql.sources.v2.bucketing.enabled=true}}, one split per row, against the 
same query with bucketing disabled:

||transform||rows||{{count(*) OVER (PARTITION BY s)}}||{{GROUP BY 
s}}||shuffles||
|{{bucket(5, s)}}|{{'A'}}, {{'a'}}|1, 1 (expected 2, 2)|2 groups of 1 (expected 
1 group of 2)|0|
|{{bucket(5, s)}}|{{U+0130}}, {{i U+0307}}|1, 1|2 groups of 1|0|
|{{prefix1(s)}}|{{U+0130}}, {{i U+0307}}|1, 1|2 groups of 1|0|
|{{identity(s)}}|either|2, 2|1 group of 2|0|

An equi-join on {{s}} is correct: it is planned over collation keys, so both 
sides shuffle.

SPARK-50593 (https://github.com/apache/spark/pull/55885) lets a transform with 
literal parameters, such as {{truncate(s, w)}}, report its partitioning, so 
{{truncate(s, 1)}} gives the same wrong rows as {{prefix1(s)}} above, where it 
shuffled before.

A fix could make a transform other than {{identity}} over a column whose type 
is not binary stable ({{UnsafeRowUtils.isBinaryStable}}) not satisfy a 
clustering on that column, e.g. in {{KeyedPartitioning.supportsExpressions}}, 
so such a scan shuffles. {{identity}} can keep its partitioning, since its keys 
are compared with the collation. A connector that knows its transform respects 
the collation has no way to say so today, so this costs those transforms their 
shuffle-free plans too.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to