Haotian Sun created SPARK-58766:
-----------------------------------
Summary: Represent PyArrow scalars Python cannot stringify in
golden-file tests
Key: SPARK-58766
URL: https://issues.apache.org/jira/browse/SPARK-58766
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 5.0.0
Reporter: Haotian Sun
The golden-file cell formatter (GoldenFileTestMixin.repr_arrow_value in
python/pyspark/testing/goldenutils.py) renders each element with str(scalar),
which
raises for PyArrow scalars that are valid in Arrow but that Python cannot turn
into an
object:
- an out-of-range temporal value (for example, int32 max cast to date32 is a
valid
date32 whose day count is ~year 5.88M, so building a Python datetime.date
raises
OverflowError), and
- non-UTF-8 bytes cast to string (str raises UnicodeDecodeError).
The conversion itself succeeds; only the string rendering fails. Today the
failure is
swallowed into a misleading ERR@<Exception> cell (which reads like a failed
conversion),
and it would crash the test once the cell try/except is narrowed to guard only
the
conversion call.
Fix: when str(scalar) raises for such a value, record the raw stored value (for
example,
raw=2147483647) instead of failing. This ticket covers the out-of-range
temporal case;
the non-UTF-8 binary case follows as a [FOLLOWUP].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
