[
https://issues.apache.org/jira/browse/SPARK-60606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60606:
----------------------------
Description:
Local input conversion builds a second complete row matrix, then clones each
column before Arrow conversion. Coerce one column at a time to remove those
intermediate copies while retaining the existing coercion rules. For
homogeneous Int64 result batches, downcast each column once and preserve the
generic decoder for all other batches.
Each optimization was measured independently against `74b8e41`; that baseline
has exactly the same source tree as current master `c36c846`. Each screen and
fresh confirmation used 15 fresh processes per build/case in rotating order.
These are observed median changes; the combined PR head has not been
benchmarked.
Selected independent confirmation observations: wide tuple input CPU −13.56%,
wall −13.49%, held RSS −28.95 MiB; string input CPU/wall −8.24%, held RSS −3.44
MiB; Int64 collect CPU −57.34%, wall −67.50%, held RSS +0.125 MiB; Int64
collect plus full value scan CPU −7.09%, wall −7.27%, held RSS +0.2344 MiB.
Overall CPU and memory acceptance rejected: the retained Duration(Microsecond)
case fails before timing in both Rust builds, and several timing regression
intervals remain uncertain. Wide-input target-only repeated memory support does
not establish overall acceptance. The combined head has no benchmark timing;
live Spark parity and CI remain outstanding.
Fresh combined correctness/API/Python validation is recorded in the draft PR.
Completion requires a fresh frozen combined-head screen and independent
confirmation with every raw Rust sample and CPU/wall/RSS guard passing, plus
live-server parity. Keep the existing Duration failure explicit; exact
timedelta conversion is separate correctness work.
Harness:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Generated with OpenAI Codex.
Draft PR: https://github.com/apache/spark-connect-rust/pull/109
was:
Local input conversion builds a second complete row matrix, then clones each
column before Arrow conversion. Coerce one column at a time to remove those
intermediate copies while retaining the existing coercion rules. For
homogeneous Int64 result batches, downcast each column once and preserve the
generic decoder for all other batches.
Each optimization was measured independently against `74b8e41`; that baseline
has exactly the same source tree as current master `c36c846`. Each screen and
fresh confirmation used 15 fresh processes per build/case in rotating order.
These are observed median changes; the combined PR head has not been
benchmarked.
Selected independent confirmation observations: wide tuple input CPU −13.56%,
wall −13.49%, held RSS −28.95 MiB; string input CPU/wall −8.24%, held RSS −3.44
MiB; Int64 collect CPU −57.34%, wall −67.50%, held RSS +0.125 MiB; Int64
collect plus full value scan CPU −7.09%, wall −7.27%, held RSS +0.2344 MiB.
Overall CPU and memory acceptance rejected: the retained Duration(Microsecond)
case fails before timing in both Rust builds, and several timing regression
intervals remain uncertain. Wide-input target-only repeated memory support does
not establish overall acceptance. The combined head has no benchmark timing;
live Spark parity and CI remain outstanding.
Fresh combined correctness/API/Python validation is recorded in the draft PR.
Completion requires a fresh frozen combined-head screen and independent
confirmation with every raw Rust sample and CPU/wall/RSS guard passing, plus
live-server parity. Keep the existing Duration failure explicit; exact
timedelta conversion is separate correctness work.
Harness:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Generated with OpenAI Codex.
> Reduce local input allocations and Int64 decoding overhead in Spark Connect
> Rust
> --------------------------------------------------------------------------------
>
> Key: SPARK-60606
> URL: https://issues.apache.org/jira/browse/SPARK-60606
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Local input conversion builds a second complete row matrix, then clones each
> column before Arrow conversion. Coerce one column at a time to remove those
> intermediate copies while retaining the existing coercion rules. For
> homogeneous Int64 result batches, downcast each column once and preserve the
> generic decoder for all other batches.
> Each optimization was measured independently against `74b8e41`; that baseline
> has exactly the same source tree as current master `c36c846`. Each screen and
> fresh confirmation used 15 fresh processes per build/case in rotating order.
> These are observed median changes; the combined PR head has not been
> benchmarked.
> Selected independent confirmation observations: wide tuple input CPU −13.56%,
> wall −13.49%, held RSS −28.95 MiB; string input CPU/wall −8.24%, held RSS
> −3.44 MiB; Int64 collect CPU −57.34%, wall −67.50%, held RSS +0.125 MiB;
> Int64 collect plus full value scan CPU −7.09%, wall −7.27%, held RSS +0.2344
> MiB.
> Overall CPU and memory acceptance rejected: the retained
> Duration(Microsecond) case fails before timing in both Rust builds, and
> several timing regression intervals remain uncertain. Wide-input target-only
> repeated memory support does not establish overall acceptance. The combined
> head has no benchmark timing; live Spark parity and CI remain outstanding.
> Fresh combined correctness/API/Python validation is recorded in the draft PR.
> Completion requires a fresh frozen combined-head screen and independent
> confirmation with every raw Rust sample and CPU/wall/RSS guard passing, plus
> live-server parity. Keep the existing Duration failure explicit; exact
> timedelta conversion is separate correctness work.
> Harness:
> https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
> Generated with OpenAI Codex.
> Draft PR: https://github.com/apache/spark-connect-rust/pull/109
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
