[
https://issues.apache.org/jira/browse/SPARK-60606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60606:
-----------------------------------
Labels: pull-request-available (was: )
> Reduce local input allocations and Int64 decoding overhead in Spark Connect
> Rust
> --------------------------------------------------------------------------------
>
> Key: SPARK-60606
> URL: https://issues.apache.org/jira/browse/SPARK-60606
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Local input conversion builds a second complete row matrix, then clones each
> column before Arrow conversion. Coerce one column at a time to remove those
> intermediate copies while retaining the existing coercion rules. For
> homogeneous Int64 result batches, downcast each column once and preserve the
> generic decoder for all other batches.
> Each optimization was measured independently against `74b8e41`; that baseline
> has exactly the same source tree as current master `c36c846`. Each screen and
> fresh confirmation used 15 fresh processes per build/case in rotating order.
> These are observed median changes; the combined PR head has not been
> benchmarked.
> Selected independent confirmation observations: wide tuple input CPU −13.56%,
> wall −13.49%, held RSS −28.95 MiB; string input CPU/wall −8.24%, held RSS
> −3.44 MiB; Int64 collect CPU −57.34%, wall −67.50%, held RSS +0.125 MiB;
> Int64 collect plus full value scan CPU −7.09%, wall −7.27%, held RSS +0.2344
> MiB.
> Overall CPU and memory acceptance rejected: the retained
> Duration(Microsecond) case fails before timing in both Rust builds, and
> several timing regression intervals remain uncertain. Wide-input target-only
> repeated memory support does not establish overall acceptance. The combined
> head has no benchmark timing; live Spark parity and CI remain outstanding.
> Fresh combined correctness/API/Python validation is recorded in the draft PR.
> Completion requires a fresh frozen combined-head screen and independent
> confirmation with every raw Rust sample and CPU/wall/RSS guard passing, plus
> live-server parity. Keep the existing Duration failure explicit; exact
> timedelta conversion is separate correctness work.
> Harness:
> https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
> Generated with OpenAI Codex.
> Draft PR: https://github.com/apache/spark-connect-rust/pull/109
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
