Ruifeng Zheng created SPARK-61452:
-------------------------------------
Summary: Cache Python datetime and decimal objects when collecting
rows
Key: SPARK-61452
URL: https://issues.apache.org/jira/browse/SPARK-61452
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Ruifeng Zheng
Collected date, timestamp, interval, and decimal values currently import their
Python modules and resolve classes for every scalar field. Cache the immutable
Python class and epoch objects once so row materialization avoids repeated
Python import-cache and module-attribute lookups.
Release microbenchmarks of the value materialization path show 1.77-3.01x
speedups, depending on the value type, while preserving the existing Python
results.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]