Ruifeng Zheng created SPARK-61452:
-------------------------------------

             Summary: Cache Python datetime and decimal objects when collecting 
rows
                 Key: SPARK-61452
                 URL: https://issues.apache.org/jira/browse/SPARK-61452
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Ruifeng Zheng


Collected date, timestamp, interval, and decimal values currently import their 
Python modules and resolve classes for every scalar field. Cache the immutable 
Python class and epoch objects once so row materialization avoids repeated 
Python import-cache and module-attribute lookups.

Release microbenchmarks of the value materialization path show 1.77-3.01x 
speedups, depending on the value type, while preserving the existing Python 
results.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to