HyukjinKwon commented on PR #59298:
URL: https://github.com/apache/spark/pull/59298#issuecomment-6095755470
Thanks @canyang25. The toInternal side looks right:
`_naive_dt_to_epoch_seconds` keeps pre-1970 microseconds exact, `datetime.min`
goes through the mktime fallback, and both local-time types share it. One gap
traces back to my earlier suggestion. The integer-second form I proposed for
`convert_timestamp` raises `ValueError: year 0 is out of range` for instants in
roughly the first day of 0001-01-01 (for example `TIMESTAMP'0001-01-01
00:00:00'` under UTC), because the local `fromtimestamp` probes the offset a
day earlier. The old `astimezone()` path returned those values, so Connect
`collect()` and Arrow UDF inputs now fail on them. Falling back to
`value.astimezone().replace(tzinfo=None)` on `(OverflowError, ValueError)`, as
the toInternal helper does, would fix it. Also, the PR description on GitHub
still describes the earlier `dt.timestamp()` /
`fromtimestamp(value.timestamp())` approach, so the update may not have saved.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:6094465541","thread_id":"conversation:6094465541","verdict_sha256":"12a6526df95d6060523b1e644a6680422edc34e5bbf73a806263de56014de191"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
