[
https://issues.apache.org/jira/browse/SPARK-60677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60677:
-----------------------------------
Labels: pull-request-available (was: )
> Cache immutable DataFrame schemas in Spark Connect Rust
> -------------------------------------------------------
>
> Key: SPARK-60677
> URL: https://issues.apache.org/jira/browse/SPARK-60677
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Repeated DataFrame.schema/columns/dtypes accesses currently perform repeated
> AnalyzePlan RPCs in the Rust Python client, whereas official PySpark Connect
> caches the schema on the DataFrame. The frozen benchmark reports repeated
> schema access at about 0.192 ms versus 0.003 ms for Python; on a
> 256-operation prepared chain Rust takes about 2.61 ms per access. Cache
> schema metadata consistently for the same immutable DataFrame plan, preserve
> isolation for derived plans and clones/session boundaries, and avoid hiding
> errors or changing public types. Add tests proving RPC counts and cache
> lifecycle, and benchmark first/repeated schema, columns, dtypes and deep-plan
> cases plus unrelated guards. Baseline Rust: d884e347452c. Session version
> caching is separate scope.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
