Spenserrrr opened a new pull request, #57830:
URL: https://github.com/apache/spark/pull/57830
### What changes were proposed in this pull request?
This PR adds a golden-file test pinning the behavior of
`pa.Array.from_pandas` with default arguments (no `mask`, no `type`, no
`safe`), under the SPARK-54936 umbrella for monitoring upstream library
behavior changes.
New file
`python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py`
with `PyArrowArrayFromPandasDefaultTests`, plus the golden files
`golden_pyarrow_array_from_pandas_default.{csv,md}` (97 rows x 2 columns) and
one line in `dev/sparktestsupport/modules.py`.
The source Series are built by three group methods, so the follow-up
non-default tests can reuse the whole inventory or one group:
- `_numpy_backed_sources()` -- numpy storage, so `hasattr(series.array,
"__arrow_array__")` is `False` and PySpark passes `mask=series.isnull()`.
- `_protocol_sources()` -- dtypes implementing `__arrow_array__`, where
PySpark passes `mask=None` because supplying a mask raises. The protocol means
"can export Arrow", not "is stored as Arrow": `Int64` is numpy values plus a
byte mask and `string[python]` is an object ndarray, yet both implement it
alongside the genuinely Arrow-backed `[pyarrow]` dtypes. PySpark does not
distinguish them -- `conversion.py:435` branches only on the protocol -- so
they share one group.
- `_coercion_sources()` -- values that are lossy or ambiguous once a target
type is requested. Their unconverted results are recorded here to give the
follow-up type tests a baseline to be read against.
A small helper spells a `ChunkedArray` result as `chunked<type>`.
`repr_value` reports the element type, which is identical for an `Array` and a
`ChunkedArray`, so without it the two are indistinguishable in a cell -- and
which one comes back is a real contract, since `create_arrow_table_from_pandas`
builds a `pa.Table` rather than a `pa.RecordBatch` precisely because a chunked
input can produce a `ChunkedArray`.
### Why are the changes needed?
`from_pandas` is the pandas -> Arrow direction, which the previously merged
golden tests under SPARK-54936 do not cover -- they are all Arrow -> pandas.
PySpark depends on it at three production call sites:
`python/pyspark/sql/conversion.py:443` and `:474` (the pandas UDF return leg,
legacy and non-legacy), and `python/pyspark/sql/pandas/conversion.py:117`
(`create_arrow_array_from_pandas`, used by `createDataFrame(pandas_df)`). A
silent change here becomes wrong data on the JVM side rather than a visible
error.
Two behaviors the golden pins that are easy to miss:
- **pandas 3 changed its default temporal resolution.**
`pd.Series([datetime])` infers `datetime64[ns]` on pandas 2 and
`datetime64[us]` on pandas 3, and `Categorical.categories` become `str` (so the
dictionary's values become `large_string`). The rows pin each resolution
explicitly and add deliberately unpinned `:inferred` rows whose only job is to
record which default pandas picks.
- **`object` dtype and `datetime64[ns]` disagree.** In object dtype pyarrow
takes the unit from `datetime.datetime`'s own microsecond resolution rather
than from a numpy dtype, so the same values give `timestamp[us]` instead of
`timestamp[ns]` -- and a sub-microsecond value is dropped silently, while an
explicit `type=pa.timestamp("us")` raises `ArrowInvalid` on that same loss.
Writing these tests is also how the `ChunkedArray` return type came to
light: `from_pandas` can return a `ChunkedArray` rather than an `Array`, which
led to tracing the two consumers and finding that `PandasToArrowConversion` did
not guard for it. That is fixed separately in #57829 (SPARK-58625); this PR
only records the upstream behavior.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
New golden-file test. It runs without a Spark session, so it exercises
PyArrow and pandas directly.
Validated across **16 environments -- PyArrow 18/19/20/21/22/23/24/25 x
pandas 2 and pandas 3** -- with a fresh virtualenv per combination, running the
committed test file against the committed golden file. All 16 pass. Six cells
legitimately differ on pandas 3 and are carried in the `overrides` dict rather
than by regenerating; disabling the `LooseVersion` guard makes pandas 3 fail on
exactly those six while pandas 2 still passes, so the overrides are
load-bearing.
Also verified: the test passes without `SPARK_GENERATE_GOLDEN_FILES`,
regeneration is byte-identical, and `ruff check` / `ruff format --check` /
`mypy` are clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
