[
https://issues.apache.org/jira/browse/SPARK-60549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-60549:
------------------------------------
Assignee: DB Tsai
> Share immutable logical plans and expressions in Spark Connect Rust
> -------------------------------------------------------------------
>
> Key: SPARK-60549
> URL: https://issues.apache.org/jira/browse/SPARK-60549
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Spark Connect Rust recursively clones existing logical-plan and expression
> subtrees when DataFrames and Columns are extended or reused. Share immutable
> logical-plan nodes through Arc<LogicalPlan> and represent expressions as
> shared Arc<ExpressionNode> handles to avoid repeatedly copying ancestors and
> payloads during client-side construction.
> Preserve structural equality (including NaN behavior), rendering, protobuf
> output, argument order, and execution-time plan-ID assignment. Include
> sharing, immutability, payload-lifetime, and serialization regressions, plus
> a reproducible native-PySpark benchmark separating construction, action, and
> total time.
> Compatibility: public Rust logical-plan child fields change from Box to Arc;
> direct expression construction and pattern matching move to ExpressionNode.
> Ordinary Column, DataFrame, and Python APIs and the Spark Connect wire
> protocol remain unchanged. Shared occurrences still serialize to independent
> protobuf trees.
> Local measurements show faster client construction, but do not establish
> consistent end-to-end speedups or memory savings. Zero-copy protobuf/Arrow
> serialization and server execution deduplication are not part of this change.
> Draft implementation: https://github.com/apache/spark-connect-rust/pull/102
> The PR records completed regression/live tests, benchmark methodology, and
> outstanding full parity/API-introspection checks and pre-existing Clippy
> failures.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
