[
https://issues.apache.org/jira/browse/SPARK-61547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-61547:
----------------------------------
Fix Version/s: connect-rust-4.3.0
> Cache the UTC epoch for local timestamp conversion
> --------------------------------------------------
>
> Key: SPARK-61547
> URL: https://issues.apache.org/jira/browse/SPARK-61547
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> infer.rs::instant_micros() imports datetime, resolves timezone.utc, and
> constructs datetime(1970, 1, 1, tzinfo=UTC) with a keyword dictionary for
> each local TimestampType value converted by createDataFrame().
> Cache the immutable UTC timezone and epoch datetime together with PyOnceLock.
> Continue calling astimezone() and datetime subtraction for each input,
> preserving local-timezone/DST handling, datetime subclass hooks, and integer
> microsecond precision. The conversion also covers timestamp values inside
> arrays, maps, and structs.
> Release-build benchmarks on CPython 3.13 measured 3.32x faster local
> createDataFrame construction for 1,000 UTC timestamps, 2.80x for 10,000 UTC
> timestamps, and 3.68x for 1,000 rows with arrays of four timestamps.
> Offset-aware and naive input improved by 2.41x and 1.64x. Integer and
> TimestampNTZ controls were effectively unchanged. These measurements exclude
> Spark execution and use a mock Connect server for metadata requests.
> Add regression coverage for reuse across calls and nested fields, timezone
> changes after cache initialization, DST folds, pre-epoch and extreme datetime
> values, microsecond precision, and subclass hook errors.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]