[
https://issues.apache.org/jira/browse/SPARK-61360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61360:
-----------------------------------
Labels: pull-request-available (was: )
> Fix DateOffset imports with pandas 3.1
> --------------------------------------
>
> Key: SPARK-61360
> URL: https://issues.apache.org/jira/browse/SPARK-61360
> Project: Spark
> Issue Type: Sub-task
> Components: Pandas API on Spark
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> The scheduled pandas 3 CI job fails before running pandas-on-Spark tests with
> pandas 3.1.1:
> https://github.com/apache/spark/actions/runs/42413930652/job/128532657680
> ImportError: cannot import name 'DateOffset' from 'pandas.tseries.frequencies'
> Both python/pyspark/pandas/frame.py and python/pyspark/pandas/series.py
> import DateOffset from pandas.tseries.frequencies. Import it from the public
> pandas.tseries.offsets namespace instead, while keeping to_offset imported
> from pandas.tseries.frequencies.
> The public import exists in the minimum-supported pandas 2.2.0. Package
> import checks passed with pandas 2.3.3 and 3.0.2, and the replacement
> DateOffset import and date-offset arithmetic passed with pandas 3.1.1. No
> version-dependent import is needed.
> This fixes the DateOffset import failure only. A full pandas-on-Spark import
> with pandas 3.1.1 then encounters the separate infer_dtype_from_object
> compatibility issue.
> Draft fix: https://github.com/apache/spark/pull/60238
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]