Anil K created SPARK-61654:
------------------------------

             Summary: Place a cogroup key stored like the layout's column by 
the column's own transform in the one-side shuffle
                 Key: SPARK-61654
                 URL: https://issues.apache.org/jira/browse/SPARK-61654
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Anil K


When a storage-partitioned layout has a cogroup side shuffled onto it 
({{spark.sql.sources.v2.bucketing.shuffle.enabled=true}}), 
{{KeyedShuffleSpec.createPartitioning}} rebuilds the layout's transform over 
the other side's key and asks whether Spark can evaluate the function on that 
key's type. For a cogroup that does not coerce its keys (e.g. a Pandas or Arrow 
cogroup), that is a stricter question than placement needs: the cogroup pairs 
groups by comparing key bytes, so a shuffled row only has to land where the 
keyed side put rows with the same bytes, which is the column's own transform on 
those bytes.

So since SPARK-60813 and SPARK-61650 
(https://github.com/apache/spark/pull/60431), a cogroup whose key is stored 
like the layout's column but is a datetime type the function would read 
differently, e.g. a {{TIME}} or {{TIMESTAMP_NTZ}} key onto a bucket over a 
{{TIMESTAMP}} column whose function declares {{TIMESTAMP}}, fails while 
planning with {{STORAGE_PARTITION_JOIN_KEY_TYPE_MISMATCH}}. The hash shuffle, 
with the flag off, runs the same query and pairs equal bytes.

A fix: when the key passes the storage check 
({{TransformExpression.keysStoredAlike}}) and is an attribute, rebuild the 
transform over that attribute retyped as the column's type. The shuffle then 
reads the key's bytes as the column's type, the call is the layout's own (which 
{{canEvaluatePartitioning}} already asserts), and the pairing equals the hash 
shuffle's for every key the storage check takes. A join's keys share the 
column's type, so joins do not change; other keys keep the current path. The 
error then fires only for keys stored otherwise, as for an {{INT}} key onto a 
{{BIGINT}} column.

This touches the code SPARK-50593 (https://github.com/apache/spark/pull/55885) 
changes: the rebuild has to replace only the transform's column slots 
({{rewriteColumnSlots}}), not its literal parameters. So it should follow 
SPARK-50593. If SPARK-60782 instead rejects cogroup keys of different types at 
analysis, this becomes moot for those types.

Found in the review of https://github.com/apache/spark/pull/60431.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to