Ruifeng Zheng created SPARK-61360:
-------------------------------------
Summary: Fix DateOffset imports with pandas 3.1
Key: SPARK-61360
URL: https://issues.apache.org/jira/browse/SPARK-61360
Project: Spark
Issue Type: Sub-task
Components: Pandas API on Spark
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
The scheduled pandas 3 CI job fails before running pandas-on-Spark tests with
pandas 3.1.1:
https://github.com/apache/spark/actions/runs/42413930652/job/128532657680
ImportError: cannot import name 'DateOffset' from 'pandas.tseries.frequencies'
Both python/pyspark/pandas/frame.py and python/pyspark/pandas/series.py import
DateOffset from pandas.tseries.frequencies. Import it from the public
pandas.tseries.offsets namespace instead, while keeping to_offset imported from
pandas.tseries.frequencies.
The public import exists in the minimum-supported pandas 2.2.0. Package import
checks passed with pandas 2.3.3 and 3.0.2, and the replacement DateOffset
import and date-offset arithmetic passed with pandas 3.1.1. No
version-dependent import is needed.
This fixes the DateOffset import failure only. A full pandas-on-Spark import
with pandas 3.1.1 then encounters the separate infer_dtype_from_object
compatibility issue.
Draft fix: https://github.com/apache/spark/pull/60238
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]