Yaoxuan Wu created SPARK-61948:
----------------------------------
Summary: ORC write/read shifts TIMESTAMPs just before a zone's
switch from local mean time by the LMT offset
Key: SPARK-61948
URL: https://issues.apache.org/jira/browse/SPARK-61948
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 3.5.8, 4.0.0
Reporter: Yaoxuan Wu
With the JVM default time zone America/Los_Angeles (e.g.
-Duser.timezone=America/Los_Angeles):
{code:python}
df = spark.sql("""SELECT * FROM VALUES
(1, TIMESTAMP'1883-11-18 19:52:58Z'),
(2, TIMESTAMP'1883-11-18 20:00:00Z') AS t(id, ts)""")
df.write.mode("overwrite").orc("/tmp/lmt")
spark.read.orc("/tmp/lmt").selectExpr("id", "unix_seconds(ts)").collect()
# expected: (1, -2717640422), (2, -2717640000)
# actual: (1, -2717640000), (2, -2717640000)
{code}
On 1883-11-18 Los Angeles switched from local mean time (-07:52:58) to PST, so
12:00:00..12:07:01 local time occurred twice. Every instant in the first pass
comes back 422 s later, on the second pass. Parquet (default CORRECTED rebase)
is correct; Parquet with LEGACY rebase shows the same shift; the DST overlap
(SPARK-31328) is correct.
Cause: ORC timestamps are always rebased to the hybrid calendar in the JVM
zone, keeping the local wall clock. {{java.time}} has LMT for Los Angeles
before the switch, while {{java.util.TimeZone}} (used for the hybrid calendar)
has -08:00 throughout, so the rebase maps both passes of the repeated wall
clock to the same instant and the read cannot recover the earlier one.
Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]