[
https://issues.apache.org/jira/browse/SPARK-59467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59467:
-----------------------------------
Labels: pull-request-available (was: )
> Timezone-naive ArrowDtype timestamps are interpreted as UTC instead of the
> session timezone in pandas conversion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59467
> URL: https://issues.apache.org/jira/browse/SPARK-59467
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Fangchen Li
> Priority: Major
> Labels: pull-request-available
>
> _check_series_convert_timestamps_internal in pyspark.sql.pandas.types only
> recognizes numpy datetime64 and pd.DatetimeTZDtype columns. A timezone-naive
> column backed by pd.ArrowDtype (for example, timestamp[us][pyarrow]) falls
> through unchanged and is interpreted as UTC rather than in the session
> timezone, so in a non-UTC session the value is shifted by the session's UTC
> offset:
> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
> ts = [datetime.datetime(2020, 1, 1, 12, 0)]
> numpy_col = pd.Series(ts, dtype="datetime64[ns]")
> arrow_col = pa.array(ts,
> pa.timestamp("us")).to_pandas(types_mapper=pd.ArrowDtype)
> spark.createDataFrame(pd.DataFrame(\{"t": numpy_col}), "t
> timestamp").collect()
> # [Row(t=datetime.datetime(2020, 1, 1, 12, 0))] -- correct
> spark.createDataFrame(pd.DataFrame(\{"t": arrow_col}), "t
> timestamp").collect()
> # [Row(t=datetime.datetime(2020, 1, 1, 4, 0))] -- off by 8 hours
> This affects createDataFrame with a TimestampType schema and pandas UDFs that
> return ArrowDtype timestamp Series, and it contradicts the documented
> behavior of spark.sql.session.timeZone for pandas timestamps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
