[
https://issues.apache.org/jira/browse/SPARK-61650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anil K updated SPARK-61650:
---------------------------
Affects Version/s: 4.3.0
(was: 5.0.0)
> Storage-partitioned join passes a datetime or interval value that its cast to
> the declared type would change by time zone or precision
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61650
> URL: https://issues.apache.org/jira/browse/SPARK-61650
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Anil K
> Priority: Major
>
> A storage-partitioned join evaluates a connector's partition transform itself
> in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes
> an argument as it is when it is stored the way the function's declared input
> type is, by comparing their physical types. SPARK-60813 refuses two datetime
> or interval types that count in different units. Two more pairs share a
> physical type and a unit, but the function still gets another value than
> Spark's cast to the declared type gives, which is what the write path
> ({{DistributionAndOrderingUtils}}) passes:
> * *another time zone reference*: {{TIMESTAMP_NTZ}} holds a local time, which
> the cast to {{TIMESTAMP}} converts with the session time zone. In
> {{America/Los_Angeles}}, the transform got 3600000000 (01:00) where the write
> path passes 32400000000 (09:00 UTC);
> * *a finer precision or end field*: the cast from {{TIME(9)}} to {{TIME(6)}}
> truncates to microseconds, and one from a {{DAY TO SECOND}} interval to a
> {{DAY TO HOUR}} one to hours, while the transform got the untruncated value,
> e.g. 3600123456789 where the write path passes 3600123456000 (with
> {{spark.sql.ansi.enabled=false}}).
> Rows of the shuffled side can then land in the wrong partitions and join
> matches are silently lost.
> A fix: let two datetime or interval types go as they are only where Spark's
> cast between them leaves the value unchanged, i.e. the same class at no finer
> a precision or end field, e.g. {{TIME(3)}} against {{TIME(6)}}, or a {{DAY TO
> HOUR}} interval against a {{DAY TO SECOND}} one, and refuse the rest, so the
> join shuffles both sides.
> SPARK-60813 kept {{TIMESTAMP_NTZ}} against {{TIMESTAMP}}, on the grounds that
> the connector computes its keys without the session time zone. Refusing it is
> safe either way: if the connector's keys come from the raw value, the join
> only shuffles both sides; if they come from the cast, as Spark's write path
> computes them, it avoids wrong results.
> Found in the review of SPARK-60813
> (https://github.com/apache/spark/pull/60431).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]