Yaoxuan Wu created SPARK-61948:
----------------------------------

             Summary: ORC write/read shifts TIMESTAMPs just before a zone's 
switch from local mean time by the LMT offset
                 Key: SPARK-61948
                 URL: https://issues.apache.org/jira/browse/SPARK-61948
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.1.1, 4.2.0, 3.5.8, 4.0.0
            Reporter: Yaoxuan Wu


With the JVM default time zone America/Los_Angeles (e.g. 
-Duser.timezone=America/Los_Angeles):
{code:python}
df = spark.sql("""SELECT * FROM VALUES
    (1, TIMESTAMP'1883-11-18 19:52:58Z'),
    (2, TIMESTAMP'1883-11-18 20:00:00Z') AS t(id, ts)""")
df.write.mode("overwrite").orc("/tmp/lmt")
spark.read.orc("/tmp/lmt").selectExpr("id", "unix_seconds(ts)").collect()
# expected: (1, -2717640422), (2, -2717640000)
# actual:   (1, -2717640000), (2, -2717640000)
{code}
On 1883-11-18 Los Angeles switched from local mean time (-07:52:58) to PST, so 
12:00:00..12:07:01 local time occurred twice. Every instant in the first pass 
comes back 422 s later, on the second pass. Parquet (default CORRECTED rebase) 
is correct; Parquet with LEGACY rebase shows the same shift; the DST overlap 
(SPARK-31328) is correct.

Cause: ORC timestamps are always rebased to the hybrid calendar in the JVM 
zone, keeping the local wall clock. {{java.time}} has LMT for Los Angeles 
before the switch, while {{java.util.TimeZone}} (used for the hybrid calendar) 
has -08:00 throughout, so the rebase maps both passes of the repeated wall 
clock to the same instant and the read cannot recover the earlier one.

Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to