[
https://issues.apache.org/jira/browse/SPARK-60539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132327#comment-18132327
]
Peter Toth commented on SPARK-60539:
------------------------------------
Let's wait for the related tickets before we decide on this.
> Storage-partitioned join places rows by a transform call whose cast depends
> on the session time zone
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-60539
> URL: https://issues.apache.org/jira/browse/SPARK-60539
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Priority: Major
>
> SPARK-60027 makes {{TransformExpression.resolvedFunction}} cast a partition
> transform's arguments to its function's {{inputTypes()}}, as the analyzer
> does for a function call. A cast that needs a time zone, such as DATE to
> TIMESTAMP, then takes the session time zone. The keys the connector reported
> do not depend on it.
> The storage-partitioned join paths that evaluate the call to place rows, the
> one-side shuffle and the identity reduce, can therefore place a row away from
> the partition the connector put its key in. Example with the test catalog's
> {{years}}, which binds a DATE but declares a TIMESTAMP, while the in-memory
> table computes the key from the DATE itself:
> * table {{t}} with a DATE column {{day}}, partitioned by {{years(day)}};
> * table {{u}} with a DATE column {{day}}, not partitioned;
> * {{spark.sql.sources.v2.bucketing.shuffle.enabled=true}} and
> {{spark.sql.session.timeZone=Asia/Tokyo}};
> * {{SELECT /*+ MERGE(t, u) */ * FROM t JOIN u ON t.day = u.day}}.
> The join drops the {{DATE'2021-01-01'}} row: the cast puts it at 2020-12-31
> 15:00 UTC, so {{years}} sends it to the 2020 partition. In UTC and
> America/Los_Angeles the same join returns every row.
> The one-side shuffle also builds the call in the task. With no explicit
> {{spark.sql.session.timeZone}}, the cast there takes the executor JVM's
> default zone, not the driver's.
> One fix: the join paths refuse a call that holds a cast which needs a time
> zone, through the {{canEvaluate}} check SPARK-60522 adds. The write path
> keeps the cast, since there the call only clusters rows.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
