dbtsai opened a new pull request, #102:
URL: https://github.com/apache/spark-connect-rust/pull/102
### What changes were proposed in this pull request?
Share immutable logical-plan and expression nodes instead of recursively
cloning
their ancestors while building DataFrames and Columns.
- Store DataFrame roots and logical-plan child edges as `Arc<LogicalPlan>`.
Transformations, grouped operations, writers, and table arguments reuse
their
existing input plans.
- Make `Expression` an immutable `Arc<ExpressionNode>` handle.
Column/expression
clones share nodes, including literal and UDF command payloads. Existing
clone
paths in functions, grouping, windows, plan lists, and Python bindings use
the
shared handles.
- Preserve structural equality, including NaN behavior, Debug/render output,
protobuf output, argument order, and execution-time plan-ID assignment.
- Add sharing/immutability/serialization regressions and a reproducible
native
PySpark benchmark with checksum validation and separate construction,
action,
and total timings. See `dev/design/plan-sharing-benchmark.md`.
This does not implement zero-copy protobuf or Arrow serialization,
deduplicate
server execution, or remove recursive serialization/destruction limits. CASE
and window containers can still copy flat collections of shared handles.
### Why are the changes needed?
Extending and reusing a plan or expression currently copies existing
subtrees.
Sharing immutable nodes avoids this repeated work. Expression and Column
clones
become O(1) with respect to subtree size.
Two local native-PySpark experiments compared separately preserved release
builds against the same Spark 4.2.0 Connect server. At depth 64, across two
rounds
with reversed client order:
| Isolated change | Workload | Construction speedup |
|---|---|---|
| Plan sharing vs upstream | Linear projection chain | 3.60-3.62x |
| Plan sharing vs upstream | Reused/branched plans | 4.56-5.07x |
| Expression sharing vs plan-sharing-only | Arithmetic expression chain |
2.40-2.61x |
| Expression sharing vs plan-sharing-only | Reused expressions | 2.78-2.88x |
These are separate experiments, not a combined-speedup measurement. Each used
10,000 rows, four branches, one input partition, three warmups, and ten
measured
repetitions per workload, at depths 16 and 64. Both experiments validated
exact
sum/count results. The expression experiment completed 320 measured actions
and
96 warmups using the actual native bindings, not the transport-injection
plugin.
End-to-end improvement was not consistent: some gains reversed with client
order. At depth 64 the expression construction savings were approximately
0.10-0.14 ms, while actions took tens of milliseconds. These retained-handle
measurements exclude final destruction and do not establish application-level
speedups or memory savings. The benchmark retains raw samples and checks
client
identity, extension hashes, runtime settings, and checksums.
### Does this PR introduce _any_ user-facing change?
Yes: direct Rust users of the public representation must migrate:
- `LogicalPlan` child fields change from `Box<LogicalPlan>` to
`Arc<LogicalPlan>`. Plan helpers accept owned/shared nodes through
`Into<Arc<LogicalPlan>>`.
- The former `Expression` enum becomes `ExpressionNode`; `Expression` is the
shared handle. For example, `Expression::Literal(value)` becomes
`Expression::new(ExpressionNode::Literal(value))`, and matching uses
`expression.node()`.
Ordinary Column, DataFrame, and Python APIs are unchanged. The Spark Connect
wire
protocol and server behavior are unchanged. Shared occurrences still expand
to
independent protobuf trees during serialization.
### How was this patch tested?
Completed on the candidate release build:
- 373 core Rust unit tests and 52 selected integration/golden tests passed
(425 total), including nine new expression-sharing tests.
- Six live Rust tests passed against Spark 4.2.0 Connect: four Column tests
and
two DataFrame tests, with one test thread.
- All core integration test targets compiled, also with the optional
`wasm-udf` feature. The native Python extension built in release mode.
- Fifteen Python benchmark-harness tests passed.
- Rust formatting and `git diff --check` passed.
- All measured and warmup benchmark results matched exact sums/counts.
The new tests check shared node identity, immutable CASE/window branching,
binary/UDF buffer lifetimes, structural equality including NaN, Debug/render
compatibility, and separate protobuf materialization of shared occurrences.
Existing plan/function/expression golden tests check unchanged serialization.
Outstanding validation, disclosed for draft review:
- Full official parity suite and API-introspection diff have not been run.
- Not all live integration suites were run.
- Workspace Clippy is not green due to pre-existing Python-binding lints and
approximate-constant test errors. Comparison/review found no new
attributable
diagnostics. This is not a claim of green Clippy or full parity.
Benchmark reproduction instructions and exact timing boundaries are in
`dev/design/plan-sharing-benchmark.md`. Use the same benchmark script and
separate release client builds, reverse client order in a second round, and
retain the raw JSON reports. Comparing upstream directly with this combined
patch measures both ownership changes together, not the expression-only
effect.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5-based agent)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
