KunalShukla-Al opened a new issue, #59824: URL: https://github.com/apache/spark/issues/59824
### What happens With pandas 3.1.0 (released 2026-10-22), `import pyspark.pandas` fails on PySpark 4.2.0 and on `master`: ``` ImportError: cannot import name 'DateOffset' from 'pandas.tseries.frequencies' ``` With that import fixed, the next one fails the same way: ``` ImportError: cannot import name 'infer_dtype_from_object' from 'pandas.core.dtypes.common' ``` `pyspark[pandas_on_spark]` requires `pandas>=2.2.0` with no upper bound, so a fresh `pip install "pyspark[pandas_on_spark]==4.2.0"` resolves pandas 3.1.0 today and the pandas API on Spark can't be imported. ### Reproduce ``` python3.12 -m venv v && . v/bin/activate pip install "pyspark==4.2.0" "pandas==3.1.0" pyarrow python -c "import pyspark.pandas" ``` With `pandas==3.0.6` the import works. ### Cause pandas 3.1.0 removed two non-public names that pandas-on-Spark imports at module level: - `python/pyspark/pandas/frame.py:64` and `python/pyspark/pandas/series.py:60`: `from pandas.tseries.frequencies import DateOffset`. The public path `from pandas.tseries.offsets import DateOffset` returns the same object as `pd.DateOffset` on pandas 2.2.3 and 3.1.0. - `python/pyspark/pandas/frame.py:70`: `from pandas.core.dtypes.common import infer_dtype_from_object`. It no longer exists in pandas 3.1.0 (`DataFrame.select_dtypes` was rewritten there), and pandas-on-Spark uses it in `DataFrame.select_dtypes`, so it needs a replacement or a local copy of the old helper. Checking every `from pandas... import` under `python/pyspark` against pandas 3.1.0 found no other breakage beyond these two, apart from optional extras (jinja2, matplotlib) and the `is_builtin_func` import in `groupby.py` that `master` already guards for pandas 3. Found with AI assistance (Claude Code); the repro was run on macOS with Python 3.12 and Java 21. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
