Spenserrrr opened a new pull request, #58178: URL: https://github.com/apache/spark/pull/58178
### What changes were proposed in this pull request? Add golden-file tests for `pa.Schema.from_pandas` in `test_pyarrow_dataframe_from_pandas.py`, alongside the existing `RecordBatch.from_pandas` tests. The test sweeps `preserve_index` (`None`/`False`/`True`) across frame shape and index kind (0-column, single/multi-column, RangeIndex, named, unnamed, and MultiIndex), pinning each inferred field's name, type, and nullability. The index-aware input rendering (`_index_desc`/`_input_cell`) is lifted from the RecordBatch class into the shared base (now a second consumer), and `GoldenFileTestMixin` gains a `pa.Schema` repr (`repr_arrow_schema_value`) that surfaces field nullability. ### Why are the changes needed? Part of SPARK-54936 (monitor upstream library behavior changes). Spark builds a `StructType` from `pa.Schema.from_pandas` at two sites that diverge on `preserve_index`: classic `createDataFrame` passes `preserve_index=False` (index dropped), while Spark Connect passes it bare/`None` (a named or non-range index becomes an extra field). Both read each field's type and nullability. Pinning this in a golden file catches an upstream behavior change loudly instead of silently altering the inferred Spark schema. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? New golden-file tests, run without a Spark session. Validated across a PyArrow 18-25 x pandas 2/3 sweep (16 combinations) with no version-specific overrides. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
