HyukjinKwon opened a new pull request, #57078: URL: https://github.com/apache/spark/pull/57078
> [!NOTE] > Draft/verification PR from a personal fork. The `[TEMP][DO-NOT-MERGE]` fork-only workflow commit (`.github/workflows/zz_fork_pandas3_verify.yml`) will be dropped before this is finalized for review. ### What changes were proposed in this pull request? This PR makes the pandas-on-Spark test suite (doctests plus a few pyarrow conversion tests) pass under **pandas 3.0**, fixing the main code where pandas-on-Spark diverged from pandas rather than papering over the differences in the tests. **Main-code fixes** - **`python/pyspark/pandas/internal.py`** (`InternalFrame.prepare_pandas_frame`): pandas 3 collapses a contiguous integer index column into a `RangeIndex` inside `DataFrame.set_index`, so an explicitly materialized index such as `ps.Index([1, 2, 3])` round-tripped to `RangeIndex(start=1, stop=4, step=1)` whereas plain pandas keeps `Index([1, 2, 3])`. A genuine *default* index is always a `RangeIndex` with `start == 0` and `step == 1`; anything else came from real index values, so we reconstruct it as a materialized `Index`. This extends the existing narrower-int-dtype reconstruction (int8/int32) to int64 under pandas 3. - **`python/pyspark/pandas/window.py`** (`RollingAndExpanding.sem`): pandas 3 computes `sem` as `std(ddof) / sqrt(count)`. We derive `std(ddof)` from the sample variance so any `ddof` is honored, and keep the pandas < 3 formula (`sample_std / sqrt(count - ddof)`) for older pandas. - **`python/pyspark/sql/pandas/types.py`** (`_to_corrected_pandas_type`) and **`python/pyspark/pandas/typedef/typehints.py`** (`spark_type_to_pandas_dtype`): gate the `StringType -> pd.StringDtype` correction on the `future.infer_string` option so it follows pandas' own switch. **Test-harness change** - **`python/pyspark/pandas/__init__.py`**: under `SPARK_TESTING`, disable `future.infer_string` when running with pandas >= 3, so the doctests keep the pandas < 3 string-dtype spelling (`object`/`None` rather than `str`/`NaN`). This is documented with a `TODO` to remove once the minimum supported pandas is >= 3, at which point the doctests should be regenerated for the new `str` dtype. **Doctest updates** (only where the difference is genuine and cannot be addressed in main code) - datetime/timedelta unit repr `[ns]` -> `[...]` via `ELLIPSIS` (pandas 3 numpy backing defaults to microsecond). - `factorize` uniques `None` -> `ELLIPSIS` (`nan` under pandas 3 for an object Index). - `# doctest: +SKIP` for `DataFrame.bool`/`Series.bool` (removed in pandas 3), `GroupBy.apply(sum)` (raises `TypeError` on int+str in pandas 3), the `Rolling/Expanding.sem` numeric examples (values legitimately differ), and the two default-`RangeIndex`-vs-`Index` cases that are indistinguishable from a default index at conversion time. **pyarrow conversion tests** made version-aware for the ns->us default, the new string dtype, and `zoneinfo`-based timezones (`.tz` no longer has `.zone`). ### Why are the changes needed? The `Build / Python-only (Python 3.12, Pandas 3)` CI lane (`build_python_3.12_pandas_3.yml`) has been failing. This makes it pass without blindly skipping tests — the main code is fixed where pandas-on-Spark diverged from pandas, and tests are only adjusted where the difference is an inherent, semantically-equivalent pandas 3 behavior change. ### Does this PR introduce _any_ user-facing change? No behavioral change other than the `internal.py` fix: under pandas 3, an explicitly materialized integer index now round-trips through `to_pandas()` as `Index([...])` instead of `RangeIndex(...)`, matching plain pandas. The two are semantically equal (`Index([1, 2, 3]).equals(RangeIndex(1, 4)) == True`). ### How was this patch tested? - Ran the pandas-on-Spark doctests and the affected pyarrow tests locally against pandas 3.0.3; all pass (the only local failures were the pre-existing `FakeSourceOne` test-DataSource classpath artifact in the local prebuilt-jars setup, which passes on CI). - Verified on GitHub Actions with the pandas 3 image (`python-312-pandas-3`). **CI results** - Passing pandas 3 run: <!-- PASS_LINK --> - Previously failing pandas 3 run (baseline, before this PR): https://github.com/apache/spark/actions/runs/28755311863 ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
