Yicong-Huang commented on code in PR #58000:
URL: https://github.com/apache/spark/pull/58000#discussion_r3782754239
##########
python/pyspark/testing/goldenutils.py:
##########
@@ -311,12 +311,33 @@ def repr_type(t: Any) -> str:
# "halffloat" -> "float16", "float" -> "float32", "double" ->
"float64"
return _ARROW_FLOAT_ALIASES.get(s, s)
+ @staticmethod
+ def _scalar_str(scalar: Any) -> str:
+ """
+ Render one PyArrow scalar for a golden cell via PyArrow's own
``str(scalar)``.
+
+ A temporal value can be valid in Arrow yet outside Python's
``datetime`` range
+ (e.g. a date32 past year 9999), where ``str`` builds a Python datetime
and
+ raises ``OverflowError``. For those, record the raw stored count as
+ ``raw=<value>`` (its unit is in the type suffix) instead of failing. A
+ non-temporal ``OverflowError`` is unexpected, so it propagates.
+ """
+ try:
+ return str(scalar).replace("\x00", "\\0")
+ except OverflowError:
+ # No Python datetime exists for this value; record the raw stored
count as
+ # ``raw=<value>``. A non-temporal overflow is unexpected, so
re-raise.
+ if pa.types.is_temporal(scalar.type):
+ return f"raw={scalar.value}"
Review Comment:
I think we are monitoring arrow's behavior for pyspark/spark, so it is valid
to say that we don't support year > 9999. maybe we can just render overflow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
