Spenserrrr opened a new pull request, #58091:
URL: https://github.com/apache/spark/pull/58091
### What changes were proposed in this pull request?
Follow-up to #58000 (SPARK-58766). That PR taught the golden-file scalar
formatter (`GoldenFileTestMixin._scalar_str` in `goldenutils.py`) to render an
out-of-range temporal scalar as the marker `temporal overflow` instead of
crashing, because a value can be valid in Arrow yet unrenderable in Python.
This PR completes that work in two parts.
**1. Binary representation.** An unsafe `binary`/`large_binary` ->
`string`/`large_string` cast relabels the underlying buffer as a string without
validating UTF-8, producing an Arrow-valid array whose non-UTF-8 elements make
`str(scalar)` raise `UnicodeDecodeError`. `_scalar_str` now catches that and
records the marker `invalid utf-8` for string-typed scalars (any other type
re-raises, mirroring the `is_temporal` guard used for `temporal overflow`). The
four affected cells in `golden_pyarrow_scalar_cast_unsafe` (`binary:standard`
and `large_binary:standard` cast to `string`/`large_string`) now render as
`[\0, invalid utf-8, hello, , None]@string` instead of `ERR@UnicodeDecodeError`.
**2. Narrowing the golden cell try/except.** In the three golden cell
helpers -- `_try_cast` (`test_pyarrow_array_cast.py`), `_to_pandas_cell`
(`test_pyarrow_arrow_to_pandas_default.py`), and `_from_pandas_cell`
(`test_pyarrow_array_from_pandas_default.py`) -- the `repr_value` formatter
call is moved outside the `try`, so only the conversion under test is guarded.
A bug in the formatter now surfaces as a real test failure rather than being
silently recorded as an `ERR@` cell. This addresses the review point carried
forward from #57939 / #58000. It is behavior-preserving for the to_pandas and
from_pandas goldens, and the cast golden passes because both
unrenderable-scalar families (temporal via #58000, binary here) now render for
real.
**A note on how to render the binary cell (for reviewers).** The fixed
marker `invalid utf-8` was chosen for consistency with `temporal overflow`. An
alternative is to render the raw bytes faithfully via
`scalar.as_buffer().to_pybytes()` (e.g. `b'\xff'`), which is also deterministic
and version-stable. Happy to switch if that reads better -- it is a one-line
change in `_scalar_str`.
### Why are the changes needed?
These golden files are drift canaries for the PyArrow behavior PySpark's
conversion layer depends on (umbrella SPARK-54936). The
`ERR@UnicodeDecodeError` cells conflated a real, deterministic upstream
behavior (an unsafe cast yielding a string that holds non-UTF-8 bytes) with a
test-formatting failure, and the both-wrapped try/except would have masked a
genuine formatter bug as an `ERR@` cell. Rendering the value for real makes the
cast behavior visible in the golden, and the narrowed try/except surfaces
formatter regressions loudly.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
- Regenerated `golden_pyarrow_scalar_cast_unsafe.{csv,md}` with
`SPARK_GENERATE_GOLDEN_FILES=1`; the diff is exactly the four
`binary`/`large_binary` -> `string`/`large_string` cells. The safe cast golden
and the nested cast goldens are unchanged (a safe binary->string cast raises
`ArrowInvalid` at cast time, so its cells stay `ERR@ArrowInvalid`).
- Swept the six affected golden test files across PyArrow 18-25 x pandas 2/3
(16 combinations): all pass, no version overrides needed.
- Ran the full `python/pyspark/tests/upstream/pyarrow/` directory and the
`python/pyspark/sql/tests/coercion/` golden UDF tests (the other consumers of
`goldenutils.py`): all pass, confirming the narrowing is inert for the pandas
paths.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
---
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
