Spenserrrr opened a new pull request, #58073: URL: https://github.com/apache/spark/pull/58073
### What changes were proposed in this pull request? This PR adds golden-file drift tests for `pa.RecordBatch.from_pandas`, under the SPARK-54936 umbrella that pins the behavior of the upstream PyArrow/pandas primitives PySpark depends on (the tests run without a Spark session and fail loudly if an upstream version bump changes a primitive). New file `python/pyspark/tests/upstream/pyarrow/test_pyarrow_dataframe_from_pandas.py` covers the `from_pandas` constructors that take a whole pandas DataFrame (RecordBatch here; `Table` / `Schema` to follow in the same file), which are distinct from `pa.Array.from_pandas` (a Series, already covered by `test_pyarrow_array_from_pandas_*`). Per-column type inference matches those Array tests, so this pins the DataFrame-level behavior instead: - whole-DataFrame assembly (several columns into one batch, combined schema); - how the pandas index is handled under `preserve_index` (`None` / `False` / `True`); and - **`num_rows` preservation for a 0-column DataFrame** — the property `createDataFrame` relies on. A test-free `_PyArrowFromPandasFrameTestBase` holds the source-frame inventory so the `Table` / `Schema` followups can reuse it. `python/pyspark/testing/goldenutils.py` gains `repr_arrow_record_batch_value` (parallel to `repr_arrow_table_value`, sharing a small `_repr_arrow_columns` helper) plus a `pa.RecordBatch` branch in `repr_value`. ### Why are the changes needed? `createDataFrame` reaches for `pa.RecordBatch.from_pandas` specifically in its 0-column branch — `python/pyspark/sql/pandas/conversion.py:1026` (classic) and `python/pyspark/sql/connect/session.py:632` (Connect) — and in the streaming state path (`python/pyspark/sql/streaming/stateful_processor_api_client.py:557`). All call it bare, relying on the default `preserve_index=None` to carry the row count of a 0-column DataFrame via the pandas index metadata. If an upstream change altered that, `createDataFrame` of a column-less relation would silently lose its rows (e.g. `count()` returning 0). This golden test catches such drift. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? New golden-file test, run without a Spark session (it exercises upstream PyArrow/pandas directly). Verified across a sweep of pyarrow 18–25 × pandas 2/3 — all 16 combinations pass against the committed golden, with no version-specific overrides needed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
