[ 
https://issues.apache.org/jira/browse/SPARK-61650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anil K updated SPARK-61650:
---------------------------
    Affects Version/s: 4.3.0
                           (was: 5.0.0)

> Storage-partitioned join passes a datetime or interval value that its cast to 
> the declared type would change by time zone or precision
> --------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61650
>                 URL: https://issues.apache.org/jira/browse/SPARK-61650
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.3.0
>            Reporter: Anil K
>            Priority: Major
>
> A storage-partitioned join evaluates a connector's partition transform itself 
> in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes 
> an argument as it is when it is stored the way the function's declared input 
> type is, by comparing their physical types. SPARK-60813 refuses two datetime 
> or interval types that count in different units. Two more pairs share a 
> physical type and a unit, but the function still gets another value than 
> Spark's cast to the declared type gives, which is what the write path 
> ({{DistributionAndOrderingUtils}}) passes:
> * *another time zone reference*: {{TIMESTAMP_NTZ}} holds a local time, which 
> the cast to {{TIMESTAMP}} converts with the session time zone. In 
> {{America/Los_Angeles}}, the transform got 3600000000 (01:00) where the write 
> path passes 32400000000 (09:00 UTC);
> * *a finer precision or end field*: the cast from {{TIME(9)}} to {{TIME(6)}} 
> truncates to microseconds, and one from a {{DAY TO SECOND}} interval to a 
> {{DAY TO HOUR}} one to hours, while the transform got the untruncated value, 
> e.g. 3600123456789 where the write path passes 3600123456000 (with 
> {{spark.sql.ansi.enabled=false}}).
> Rows of the shuffled side can then land in the wrong partitions and join 
> matches are silently lost.
> A fix: let two datetime or interval types go as they are only where Spark's 
> cast between them leaves the value unchanged, i.e. the same class at no finer 
> a precision or end field, e.g. {{TIME(3)}} against {{TIME(6)}}, or a {{DAY TO 
> HOUR}} interval against a {{DAY TO SECOND}} one, and refuse the rest, so the 
> join shuffles both sides.
> SPARK-60813 kept {{TIMESTAMP_NTZ}} against {{TIMESTAMP}}, on the grounds that 
> the connector computes its keys without the session time zone. Refusing it is 
> safe either way: if the connector's keys come from the raw value, the join 
> only shuffles both sides; if they come from the cast, as Spark's write path 
> computes them, it avoids wrong results.
> Found in the review of SPARK-60813 
> (https://github.com/apache/spark/pull/60431).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to