[
https://issues.apache.org/jira/browse/SPARK-61316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61316:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Encode datetime literals exactly, as PySpark's TimestampType does
> -----------------------------------------------------------------
>
> Key: SPARK-61316
> URL: https://issues.apache.org/jira/browse/SPARK-61316
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The drop-in computes the microseconds of a datetime (lit() and
> createDataFrame) from a float of seconds, `datetime.timestamp() * 1e6`,
> rounded. The float loses the microseconds after the year 2255, so
> `F.lit(datetime(2300, 1, 1, 0, 0, 0, 123457, tzinfo=UTC))` sends .123456, and
> `F.lit(datetime(9999, 12, 31, 23, 59, 59, 999999))` raises OverflowError.
> Official PySpark 4.3.0 computes the seconds as an integer (calendar.timegm of
> the UTC tuple for an aware datetime, time.mktime of the local tuple for a
> naive one) and adds the microseconds, so the values are exact from year 1 to
> year 9999.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
