Spenserrrr opened a new pull request, #58056: URL: https://github.com/apache/spark/pull/58056
### What changes were proposed in this pull request? Add golden-file tests for `pa.Table.cast(target_schema)` under the SPARK-54936 umbrella (monitor upstream PyArrow/pandas behavior). New file `python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_cast.py` with `PyArrowTableCastTests`, covering both `safe=True` (default) and `safe=False` via separate golden files, plus a `dev/sparktestsupport/modules.py` registration. ### Why are the changes needed? PySpark's conversion layer relies on `pa.Table.cast(schema)` in `DataFrame.toArrow()` and `createDataFrame()` (both classic and Spark Connect). Unlike `pa.Array.cast` (already covered by `test_pyarrow_array_cast.py`), `Table.cast` pins genuinely Table-level behavior that the Array-level tests cannot: - multi-column, whole-schema casts; - strict positional + name-equal column matching (a name mismatch, a pure reorder, or a wrong field count all raise `ValueError` — which is why Spark Connect pre-aligns names via `rename_columns()` before `cast()`); - target-field nullability enforcement; - temporal unit/timezone coercion; - the empty-table edges (0 columns / 0 rows). These tests fail loudly if an upstream version changes that behavior, instead of PySpark silently returning wrong data. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? New golden-file tests, run without a Spark session. Verified across PyArrow 18-25 x pandas 2/3 (16/16 combinations); regenerated goldens are byte-identical and no version-specific overrides are needed (`Table.cast` is pandas-free). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
