Spenserrrr opened a new pull request, #57909: URL: https://github.com/apache/spark/pull/57909
### What changes were proposed in this pull request? This is a follow-up to SPARK-58656, which centralized the `to_pandas` golden test inventory into a shared `_PyArrowToPandasTestBase`. It extends that inventory to cover `pa.ChunkedArray.to_pandas` across the existing golden tests, plus the memory-tuning arguments that only apply on the chunked (Table->columns) path. Concretely: - Adds a `_chunked_sources()` group method to the shared base covering the chunk-count axis (zero-chunk, empty-chunk, single-chunk, and multi-chunk variants) across representative types (int64, float64, string, list, struct). Because the base's `_build_source_arrays()` unions all groups, these rows flow into the default, zero_copy, zero_copy_arrow_backed, and integer_object_nulls goldens automatically, so each records how chunking interacts with its argument. - Consolidates the chunk rows that previously lived ad hoc in the `ZeroCopy` and `IntegerObjectNulls` subclasses into the shared group (the moved cells are value-identical; the `ZeroCopy` slice rows, which are genuinely zero-copy-specific, stay local). - Adds `PyArrowChunkedArrayToPandasMemoryFlagsTests`, pinning `to_pandas` under the `self_destruct` / `split_blocks` / `use_threads` bundle that `toPandas()` uses. The three are recorded as one column because Spark only ever sets them together, and a final column records that the source ChunkedArray stays readable after `self_destruct=True` (freeing is an optimization, not a contract). - Adds chunked `timestamp[us]` rows to the coerce_temporal test, where per-chunk unit coercion is uniquely observable. Spark converts a `pa.ChunkedArray` (not a plain Array) on the Table->columns path in `python/pyspark/sql/pandas/conversion.py` (`df.toPandas()`, the empty-dataset branch, and the pandas-UDF paths after `combine_chunks()`), so these tests pin the upstream behavior PySpark depends on there. ### Why are the changes needed? These golden tests act as a drift canary for upstream PyArrow/pandas behavior under umbrella SPARK-54936. `pa.ChunkedArray.to_pandas` was previously covered only incidentally by a few scattered rows; this makes the coverage systematic so a behavior change in a future PyArrow/pandas release fails CI loudly instead of silently changing what PySpark returns. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? New and updated golden-file tests, verified across a sweep of PyArrow 18-25 x pandas 2 and 3 (16 combinations), all passing. Existing goldens whose row set changed were regenerated and diffed to confirm the moved chunk cells are value-identical. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
