DB Tsai created SPARK-60549:
-------------------------------
Summary: Share immutable logical plans and expressions in Spark
Connect Rust
Key: SPARK-60549
URL: https://issues.apache.org/jira/browse/SPARK-60549
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
Spark Connect Rust recursively clones existing logical-plan and expression
subtrees when DataFrames and Columns are extended or reused. Share immutable
logical-plan nodes through Arc<LogicalPlan> and represent expressions as shared
Arc<ExpressionNode> handles to avoid repeatedly copying ancestors and payloads
during client-side construction.
Preserve structural equality (including NaN behavior), rendering, protobuf
output, argument order, and execution-time plan-ID assignment. Include sharing,
immutability, payload-lifetime, and serialization regressions, plus a
reproducible native-PySpark benchmark separating construction, action, and
total time.
Compatibility: public Rust logical-plan child fields change from Box to Arc;
direct expression construction and pattern matching move to ExpressionNode.
Ordinary Column, DataFrame, and Python APIs and the Spark Connect wire protocol
remain unchanged. Shared occurrences still serialize to independent protobuf
trees.
Local measurements show faster client construction, but do not establish
consistent end-to-end speedups or memory savings. Zero-copy protobuf/Arrow
serialization and server execution deduplication are not part of this change.
Draft implementation: https://github.com/apache/spark-connect-rust/pull/102
The PR records completed regression/live tests, benchmark methodology, and
outstanding full parity/API-introspection checks and pre-existing Clippy
failures.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
