Yicong-Huang commented on code in PR #58000:
URL: https://github.com/apache/spark/pull/58000#discussion_r3780354702
##########
python/pyspark/testing/goldenutils.py:
##########
@@ -311,12 +311,33 @@ def repr_type(t: Any) -> str:
# "halffloat" -> "float16", "float" -> "float32", "double" ->
"float64"
return _ARROW_FLOAT_ALIASES.get(s, s)
+ @staticmethod
+ def _scalar_str(scalar: Any) -> str:
+ """
+ Render one PyArrow scalar for a golden cell via PyArrow's own
``str(scalar)``.
+
+ A temporal value can be valid in Arrow yet outside Python's
``datetime`` range
+ (e.g. a date32 past year 9999), where ``str`` builds a Python datetime
and
+ raises ``OverflowError``. For those, record the raw stored count as
+ ``raw=<value>`` (its unit is in the type suffix) instead of failing. A
+ non-temporal ``OverflowError`` is unexpected, so it propagates.
+ """
+ try:
+ return str(scalar).replace("\x00", "\\0")
+ except OverflowError:
+ # No Python datetime exists for this value; record the raw stored
count as
+ # ``raw=<value>``. A non-temporal overflow is unexpected, so
re-raise.
+ if pa.types.is_temporal(scalar.type):
+ return f"raw={scalar.value}"
Review Comment:
Two questions:
1. for upstreaming mornitoring purpose, we want to write this into a golden
file, so we can choose a format to represent it. however currently `raw` does
not read as a timestamp. maybe a format like `temporal[raw](xxxx)` or
`raw-temporal=xxxx` would be better?
2. eventually the time value has to be processed in python, using python
datatime, pandas to_datetime etc. Also values more than 9999 year are not
really meaningful anyway IMO. can we check on the scala side what is the range
for datetime? if year 9999 is a valid upper bound for spark, we might not need
to care about arrow's loose bound?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
