Ruifeng Zheng created SPARK-61360:
-------------------------------------

             Summary: Fix DateOffset imports with pandas 3.1
                 Key: SPARK-61360
                 URL: https://issues.apache.org/jira/browse/SPARK-61360
             Project: Spark
          Issue Type: Sub-task
          Components: Pandas API on Spark
    Affects Versions: 5.0.0
            Reporter: Ruifeng Zheng


The scheduled pandas 3 CI job fails before running pandas-on-Spark tests with 
pandas 3.1.1:

https://github.com/apache/spark/actions/runs/42413930652/job/128532657680

ImportError: cannot import name 'DateOffset' from 'pandas.tseries.frequencies'

Both python/pyspark/pandas/frame.py and python/pyspark/pandas/series.py import 
DateOffset from pandas.tseries.frequencies. Import it from the public 
pandas.tseries.offsets namespace instead, while keeping to_offset imported from 
pandas.tseries.frequencies.

The public import exists in the minimum-supported pandas 2.2.0. Package import 
checks passed with pandas 2.3.3 and 3.0.2, and the replacement DateOffset 
import and date-offset arithmetic passed with pandas 3.1.1. No 
version-dependent import is needed.

This fixes the DateOffset import failure only. A full pandas-on-Spark import 
with pandas 3.1.1 then encounters the separate infer_dtype_from_object 
compatibility issue.

Draft fix: https://github.com/apache/spark/pull/60238




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to