DB Tsai created SPARK-60585:
-------------------------------
Summary: Share immutable field names across decoded rows in the
Rust Spark Connect client
Key: SPARK-60585
URL: https://issues.apache.org/jira/browse/SPARK-60585
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
Arrow row decoding in the Rust Spark Connect client clones every column-name
String for every row, even though all rows in a record batch have the same
field names. Wide collected results repeat this allocation and copying work.
Share immutable field names once per Arrow record batch through a private
owned/shared representation. Keep locally constructed rows on the existing
owned Vec<String> path. Preserve field order, duplicate-name lookup, value
ownership, content equality, Debug output and the public Rust/Python Row
interfaces.
Draft implementation and benchmark details:
https://github.com/apache/spark-connect-rust/pull/105
Historical screening on the PR #102-based prototype (baseline
ff6fddd3afce376895c802a5a0a113105060a1e8) observed approximately 47–54% lower
CPU and 83–234 MiB lower held process RSS in the wide collect cases. Fully
reading every value in the 512-column result observed approximately 14% lower
CPU. These are selected prototype measurements, not measurements of the current
master-based PR head. RSS includes warmups and allocator history; it is not
allocated bytes saved per call. The fixture returns canned localhost gRPC/Arrow
responses without a Spark execution engine.
Overall benchmark acceptance remains inconclusive: construction/input timing
guards are unresolved, and the owned/shared representation has no independent
confirmation run. Before calling the optimization ready, run a fresh comparison
on the current base, confirm the result independently, assess all CPU/wall/RSS
guards, and complete broader/live-server parity checks.
The rebased implementation passed 398 Rust checks, release integration-test
compilation with and without wasm-udf, formatting, core Clippy and a Python API
snapshot comparison covering 108 exported classes. Workspace Clippy is limited
by the missing wasm target; live-server parity remains outstanding.
Generated-by: Isaac (OpenAI Codex, GPT-6).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
