DB Tsai created SPARK-60606:
-------------------------------
Summary: Reduce local input allocations and Int64 decoding
overhead in Spark Connect Rust
Key: SPARK-60606
URL: https://issues.apache.org/jira/browse/SPARK-60606
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
Local input conversion builds a second complete row matrix, then clones each
column before Arrow conversion. Coerce one column at a time to remove those
intermediate copies while retaining the existing coercion rules. For
homogeneous Int64 result batches, downcast each column once and preserve the
generic decoder for all other batches.
Each optimization was measured independently against `74b8e41`; that baseline
has exactly the same source tree as current master `c36c846`. Each screen and
fresh confirmation used 15 fresh processes per build/case in rotating order.
These are observed median changes; the combined PR head has not been
benchmarked.
Selected independent confirmation observations: wide tuple input CPU −13.56%,
wall −13.49%, held RSS −28.95 MiB; string input CPU/wall −8.24%, held RSS −3.44
MiB; Int64 collect CPU −57.34%, wall −67.50%, held RSS +0.125 MiB; Int64
collect plus full value scan CPU −7.09%, wall −7.27%, held RSS +0.2344 MiB.
Overall CPU and memory acceptance rejected: the retained Duration(Microsecond)
case fails before timing in both Rust builds, and several timing regression
intervals remain uncertain. Wide-input target-only repeated memory support does
not establish overall acceptance. The combined head has no benchmark timing;
live Spark parity and CI remain outstanding.
Fresh combined correctness/API/Python validation is recorded in the draft PR.
Completion requires a fresh frozen combined-head screen and independent
confirmation with every raw Rust sample and CPU/wall/RSS guard passing, plus
live-server parity. Keep the existing Duration failure explicit; exact
timedelta conversion is separate correctness work.
Harness:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Generated with OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
