Spenserrrr opened a new pull request, #57862: URL: https://github.com/apache/spark/pull/57862
### What changes were proposed in this pull request? This PR adds a golden-file test pinning the behavior of `pa.Array.from_pandas()` under the `mask` argument, part of the umbrella effort ([SPARK-54936](https://issues.apache.org/jira/browse/SPARK-54936)) to monitor upstream PyArrow/pandas behavior that PySpark's pandas <-> Arrow conversion layer depends on. These tests run without a Spark session and act as a drift canary: if a library upgrade changes the primitive's behavior, the golden comparison fails loudly at the exact cell. The changes: - New `test_pyarrow_array_from_pandas_non_default.py` with `PyArrowArrayFromPandasMaskTests`, which subclasses `_PyArrowFromPandasTestBase` (from the default test) to reuse its source Series inventory, and records `golden_pyarrow_array_from_pandas_mask.{csv,md}` (98 rows x 3 columns: `pandas series | mask=None | mask=isnull()`). - A small behavior-preserving refactor of the merged default test, `test_pyarrow_array_from_pandas_default.py`: - the source-Series inventory and `repr_from_pandas_result` were already lifted into a test-free base `_PyArrowFromPandasTestBase`, with `PyArrowArrayFromPandasDefaultTests` as a thin subclass, so the non-default tests can subclass the base directly (no unbound-call trick, and the base carries no `test_*` to be re-collected); - the shared `_from_pandas_cell(series, **kwargs)` cell formatter is now a method on that base, used by both the default and the new mask test. - Registered the new module in `dev/sparktestsupport/modules.py`. PySpark does not pass `mask` freely; it derives it from how the Series is stored (`python/pyspark/sql/conversion.py:435`, `python/pyspark/sql/pandas/conversion.py:113`): ```python mask = None if hasattr(series.array, "__arrow_array__") else series.isnull() ``` The golden records both `mask=None` and `mask=isnull()` so the two regimes stay observable: - **numpy-backed** dtypes do not implement `__arrow_array__`, so PySpark passes `mask=series.isnull()`. `from_pandas` also infers nulls from the Series at `mask=None`, so the two columns agree; the test pins that agreement. - **protocol** dtypes (implementing `__arrow_array__` -- the nullable extension, `string[python]`, and `[pyarrow]` dtypes) return a finished Arrow array with their own validity bitmap, so PyArrow rejects a caller-supplied mask with `ValueError` on the argument's mere presence (even an all-False no-op mask raises). PySpark passes `mask=None` for them, and `mask=isnull()` records `ERR@ValueError`. Because `mask` is fixed by the input row's dtype, it is a column pair rather than an independent matrix dimension. ### Why are the changes needed? PySpark calls `pa.Array.from_pandas(series, mask=mask, ...)` on the pandas -> Arrow path (`createDataFrame(pandas_df)` and every pandas UDF's return value). The `mask` argument's two-regime behavior -- silently accepted on numpy-backed input, rejected on protocol input -- is upstream behavior PySpark relies on but does not itself test. A golden test makes any future drift (for example, protocol dtypes beginning to accept a mask, or `mask=None` no longer inferring nulls) fail visibly in CI instead of silently changing conversion results. ### Does this PR introduce _any_ user-facing change? No. This adds tests only. ### How was this patch tested? New golden-file test, run with and without `SPARK_GENERATE_GOLDEN_FILES=1` (regeneration is byte-identical). Validated across the full support matrix -- PyArrow 18, 19, 20, 21, 22, 23, 24, 25 x pandas 2 and 3 (16/16 combinations pass), each in a fresh virtualenv running the committed test against the committed golden. `ruff check` and `ruff format --check` are clean. The pandas-3 differences are recorded as version-guarded `overrides` (notably `string:inferred`, whose dtype becomes the dedicated `str` type on pandas 3 and therefore moves from a mask-accepting row to a protocol row). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Anthropic), model Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
