Spenserrrr opened a new pull request, #58000: URL: https://github.com/apache/spark/pull/58000
### What changes were proposed in this pull request? The golden-file cell formatter `GoldenFileTestMixin.repr_arrow_value` (in `python/pyspark/testing/goldenutils.py`) renders each element with `str(scalar)`. This raises for a PyArrow temporal value that is valid in Arrow but outside Python's `datetime` range: for example, `int32` max cast to `date32` is a valid `date32` whose day count lands around year 5.88M, so building a Python `datetime.date` raises `OverflowError`. The cast succeeds; only the string rendering fails. This PR extracts a `_scalar_str` helper (used by both `repr_arrow_value` and `repr_arrow_table_value`) that, on `OverflowError`, records the raw stored value as `raw=<value>` when the scalar is temporal (`pa.types.is_temporal`), and re-raises otherwise. The two affected cast golden cells (`int32:max_min` x `date32`, in the safe and unsafe goldens) change from `ERR@OverflowError` to `[raw=2147483647, raw=-2147483648, None]@date32[day]`. This was surfaced while following up on the review of #57939 (SPARK-58720), which suggested narrowing the golden cell `try/except` to guard only the conversion call: with the formatter no longer raising on a valid-but-unrenderable value, that narrowing becomes safe. The analogous non-UTF-8 `binary` -> `string` case (which raises `UnicodeDecodeError`), together with the narrowing itself, will follow as a separate follow-up. ### Why are the changes needed? `ERR@OverflowError` reads like a failed conversion, but the conversion actually succeeded and produced a valid `date32`; only Python's `str()` cannot render it. Recording the raw stored value keeps the golden honest and reserves `ERR@` for genuine conversion failures. It also lets the cell `try/except` be narrowed to the conversion (a follow-up) without a successful-but-unrenderable value crashing the test. ### Does this PR introduce _any_ user-facing change? No. This changes test infrastructure only (golden-file test helpers and their golden files). ### How was this patch tested? Existing golden-file tests, with the two affected cells regenerated. Verified across PyArrow 18-25 x pandas 2/3 (16 combinations), all passing; the affected scalar cast matrix runs wherever numpy >= 2.0. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac --- This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
