canyang25 opened a new pull request, #59298: URL: https://github.com/apache/spark/pull/59298
### What changes were proposed in this pull request? Python UDF timestamp handling dropped datetime.fold during local-time conversion in the repeated DST hour, causing fold=1 values to be observed one hour early. Two changes: - python/pyspark/sql/conversion.py: Arrow UDF input conversion now uses datetime.datetime.fromtimestamp(value.timestamp()) instead of value.astimezone().replace(tzinfo=None), preserving fold. - python/pyspark/sql/types.py: TimestampType.toInternal (naive datetime path) now uses dt.timestamp() instead of time.mktime(dt.timetuple()), making epoch conversion fold-aware. Aware-datetime handling is unchanged. ### Why are the changes needed? When clocks go back for DST, one local hour happens twice; Python marks the second occurrence with datetime.fold=1. PySpark dropped fold in these two spots, so a TIMESTAMP in the second occurrence arrived at Python UDFs one hour too early — even an identity UDF changed the value. Since 4.2 made Arrow UDFs the default, every Python UDF reads such timestamps wrongly by default. ### Does this PR introduce _any_ user-facing change? Yes — bug fix: Python UDFs now receive the correct instant for timestamps in the repeated DST hour. Non-ambiguous timestamps are unaffected. ### How was this patch tested? New regression test test_udf_timestamp_dst_fold_round_trip in python/pyspark/sql/tests/test_udf.py: with TZ=America/Los_Angeles and spark.sql.session.timeZone=America/Los_Angeles, timestamp'2021-11-07 09:30:00Z' (epoch 1636277400, fold=1) must arrive at a Python UDF as 1636277400 on BOTH the Arrow path (default) and useArrow=False; identity UDF round-trip returns the same instant; control case for the first 01:30 (fold=0) stays 1636273800. Targeted PySpark UDF tests were run and pass. JIRA: https://issues.apache.org/jira/browse/SPARK-60081 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
