Yaoxuan Wu created SPARK-60081:
----------------------------------
Summary: Python UDFs get the wrong value for TIMESTAMPs in the
repeated DST hour
Key: SPARK-60081
URL: https://issues.apache.org/jira/browse/SPARK-60081
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.1.1, 4.2.0, 5.0.0
Reporter: Yaoxuan Wu
When clocks go back for DST, one local hour happens twice. For a TIMESTAMP in
the second occurrence of that hour, a Python UDF gets a value that is one hour
too early. Even an identity UDF {{lambda x: x}} returns a different value.
{code:java}
import os, time
os.environ["TZ"] = "America/Los_Angeles"
time.tzset()
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.types import LongType, TimestampType
spark = SparkSession.builder.config("spark.sql.session.timeZone",
"America/Los_Angeles").getOrCreate()
# 2021-11-07 09:30 UTC = 01:30 PST, the second 01:30 of the night the clocks go
back.
# Its epoch second is 1636277400.
df = spark.sql("SELECT timestamp'2021-11-07 09:30:00Z' AS t")# 1) The value a
Python UDF receives
udf_input = F.udf(lambda x: int(x.timestamp()), LongType())
df.select(udf_input("t")).first()[0]
# expected: 1636277400
# Spark 4.2.0: 1636273800 (one hour early)# 2) An identity UDF should return
its input unchanged
identity = F.udf(lambda x: x, TimestampType())
df.select(F.unix_timestamp(identity("t"))).first()[0]
# expected: 1636277400
# Spark 4.2.0: 1636273800 (one hour early)
{code}
The results above use 4.2.0's default, an Arrow-optimized UDF. Both paths are
wrong on 4.1.1, 4.2.0 and master:
|| ||What the UDF receives||What an identity UDF returns||
|Arrow UDF ({{{}useArrow=True{}}}, default in 4.2)|1 hour early|1 hour early|
|Pickled UDF ({{{}useArrow=False{}}}, default in 4.1)|correct|1 hour early|
The first 01:30 (PDT, {{{}08:30Z{}}}) and all other times are correct.
*Cause*
Python marks the second occurrence with {{{}datetime.fold = 1{}}}, and PySpark
drops {{fold}} in two places:
# Arrow input conversion ({{{}pyspark/sql/conversion.py{}}}):
{{value.astimezone().replace(tzinfo=None)}} always gives {{{}fold=0{}}}. A fix:
{{{}datetime.datetime.fromtimestamp(value.timestamp()){}}}.
# {{TimestampType.toInternal}} ({{{}pyspark/sql/types.py{}}}, used for UDF
results on the pickled path and by {{{}createDataFrame{}}}):
{{time.mktime(dt.timetuple())}} ignores {{{}fold{}}}. A fix:
{{{}dt.timestamp(){}}}.
Since pyspark-4.2 makes Arrow UDFs the default, every Python UDF now reads
these timestamps wrongly by default, while the pickled path reads them
correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
