[
https://issues.apache.org/jira/browse/SPARK-60683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60683:
-----------------------------------
Labels: pull-request-available (was: )
> Defer SQL function error imports in Spark Connect Rust
> ------------------------------------------------------
>
> Key: SPARK-60683
> URL: https://issues.apache.org/jira/browse/SPARK-60683
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Fresh whole-plan construction is dominated by the first pyspark.sql.functions
> import. A local cProfile of the frozen Rust master attributes nearly all
> first-call time to importing pyspark.errors and its JSON error catalog,
> reached by the module-level higher-order function PySparkValueError import.
> Ordinary warm union construction takes microseconds. Defer error-only imports
> in the custom SQL functions facade while preserving higher-order function
> exception classes, conditions and messages. Validate unchanged fresh-process
> construction benchmarks and both valid and invalid higher-order function
> guards. Do not alter vendored error modules or simply move the import cost to
> startup.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
