Spenserrrr opened a new pull request, #57624: URL: https://github.com/apache/spark/pull/57624
### What changes were proposed in this pull request? This PR adds a golden-file test that monitors the zero-copy behavior of `pa.Array.to_pandas(zero_copy_only=True)`, under the SPARK-54936 umbrella for detecting upstream PyArrow/pandas behavior drift. It adds `PyArrowArrayToPandasZeroCopyTests` as a new subclass in the existing `python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py` (reusing the shared `_PyArrowToPandasTestBase` and the centralized golden-matrix driver), plus a `_numpy_shares_arrow_buffer` helper on the base class. The golden file records two things per source Arrow array: - **`zero_copy_only=True`**: PyArrow's own verdict for whether the Arrow -> pandas conversion is zero-copy (`Series[dtype]`) or requires a copy (`ERR@ArrowInvalid`). - **`numpy shares arrow buffer`**: an *independent* verification (via `np.shares_memory`) of whether the default `to_pandas()` result physically shares memory with a source Arrow buffer. This does not simply trust PyArrow's flag; it measures the actual buffer sharing. The row set targets the layout properties that determine zero-copy -- no-null vs null primitives (across integer widths and float), bool, string/binary, temporal types across units, sliced (offset) arrays, and single- vs multi-chunk ChunkedArrays -- rather than re-enumerating every Arrow type (that is already covered by `test_pyarrow_arrow_to_pandas_default.py`). **Scope note:** this PR intentionally covers the commonly-used NumPy-backed `zero_copy_only` path. Testing the Arrow-backed path (`types_mapper=pd.ArrowDtype`), and whether/how to verify zero-copy independently of PyArrow's flag across the pandas 2 -> 3 transition, are open design questions being discussed separately before finalizing. Opened as a draft to make the code diff visible for that discussion. ### Why are the changes needed? PySpark relies on `pa.Array.to_pandas` throughout its conversion layer (`python/pyspark/sql/pandas/conversion.py`), and whether a given Arrow type converts to pandas without copying its buffers directly affects the memory and latency of `toPandas` and pandas UDFs. Pinning this behavior in a golden file lets CI fail loudly if it drifts across PyArrow/pandas/NumPy upgrades, instead of silently regressing performance. ### Does this PR introduce _any_ user-facing change? No. This is a test-only change. ### How was this patch tested? This is itself a test. It runs without a Spark session (it exercises PyArrow/pandas directly). Verified both with `SPARK_GENERATE_GOLDEN_FILES=1` (regenerating and eyeballing the golden file) and without the flag (comparing against the committed golden file), on pandas 2.3.3, pyarrow 24.0.0, numpy 2.4.1. ruff check + format pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
