Ruifeng Zheng created SPARK-61547:
-------------------------------------
Summary: Cache the UTC epoch for local timestamp conversion
Key: SPARK-61547
URL: https://issues.apache.org/jira/browse/SPARK-61547
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Ruifeng Zheng
infer.rs::instant_micros() imports datetime, resolves timezone.utc, and
constructs datetime(1970, 1, 1, tzinfo=UTC) with a keyword dictionary for each
local TimestampType value converted by createDataFrame().
Cache the immutable UTC timezone and epoch datetime together with PyOnceLock.
Continue calling astimezone() and datetime subtraction for each input,
preserving local-timezone/DST handling, datetime subclass hooks, and integer
microsecond precision. The conversion also covers timestamp values inside
arrays, maps, and structs.
Release-build benchmarks on CPython 3.13 measured 3.32x faster local
createDataFrame construction for 1,000 UTC timestamps, 2.80x for 10,000 UTC
timestamps, and 3.68x for 1,000 rows with arrays of four timestamps.
Offset-aware and naive input improved by 2.41x and 1.64x. Integer and
TimestampNTZ controls were effectively unchanged. These measurements exclude
Spark execution and use a mock Connect server for metadata requests.
Add regression coverage for reuse across calls and nested fields, timezone
changes after cache initialization, DST folds, pre-epoch and extreme datetime
values, microsecond precision, and subclass hook errors.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]