[
https://issues.apache.org/jira/browse/SPARK-60585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60585:
----------------------------
Description:
Arrow row decoding repeatedly copies column names even though all rows in a
record batch have the same names. Share immutable top-level names through
Arc<[String]>, retaining owned names for locally constructed rows. Catalog and
DataFrame use one common decoder. Nested Value::Struct names still copy per row
and are outside this change.
Implementation and benchmark table:
https://github.com/apache/spark-connect-rust/pull/105
Fresh comparison: Apache master a85e474f13b958fd3bd5ba7f91055dd2f2b959f3
against PR head 6dccb5716d6bc52dd11beab061216ea034f7ad51. Independent screening
and confirmation each measured 15 cases with 30 fresh processes per
client/case: 2,700 successful samples total, zero failures. Source, release
wheel/native, harness and policy identities were verified before and after
measurement.
Selected confirmation medians, baseline to candidate: 100,000 x 50 collect CPU
238.56 to 129.14 ms, wall 230.43 to 119.85 ms, held RSS 680.44 to 446.91 MiB;
4,096 x 512 collect CPU 89.31 to 41.17 ms, wall 84.70 to 37.02 ms, held RSS
297.89 to 214.59 MiB. The fully scanned 4,096 x 512 case measured CPU 355.05 to
308.66 ms. RSS is process memory including allocator history. Canned Arrow/gRPC
fixtures do not execute Spark plans.
Overall CPU and memory acceptance remain inconclusive. All RSS regression
guards passed in both runs, but timing intervals for expression/type
construction, toArrow and catalog RPC guards remain uncertain under the
unchanged policy. Further validation is required; selected case improvements do
not override these guards.
399 Rust checks passed; all core integration targets compiled with and without
wasm-udf. The 108-class Python API snapshot is unchanged. Formatting and core
Clippy passed. Row remains 48 bytes, with the private RowFields representation
24 bytes on this 64-bit build. Local workspace Clippy lacks the wasm target.
Green CI and live-server parity remain outstanding.
Harness: https://github.com/dbtsai/pyspark-connect-benchmark
Generated-by: Isaac (OpenAI Codex, GPT-6).
was:
Arrow row decoding in the Rust Spark Connect client clones every column-name
String for every row, even though all rows in a record batch have the same
field names. Wide collected results repeat this allocation and copying work.
Share immutable field names once per Arrow record batch through a private
owned/shared representation. Keep locally constructed rows on the existing
owned Vec<String> path. Preserve field order, duplicate-name lookup, value
ownership, content equality, Debug output and the public Rust/Python Row
interfaces.
Draft implementation and benchmark details:
https://github.com/apache/spark-connect-rust/pull/105
Historical screening on the PR #102-based prototype (baseline
ff6fddd3afce376895c802a5a0a113105060a1e8) observed approximately 47–54% lower
CPU and 83–234 MiB lower held process RSS in the wide collect cases. Fully
reading every value in the 512-column result observed approximately 14% lower
CPU. These are selected prototype measurements, not measurements of the current
master-based PR head. RSS includes warmups and allocator history; it is not
allocated bytes saved per call. The fixture returns canned localhost gRPC/Arrow
responses without a Spark execution engine.
Overall benchmark acceptance remains inconclusive: construction/input timing
guards are unresolved, and the owned/shared representation has no independent
confirmation run. Before calling the optimization ready, run a fresh comparison
on the current base, confirm the result independently, assess all CPU/wall/RSS
guards, and complete broader/live-server parity checks.
The rebased implementation passed 398 Rust checks, release integration-test
compilation with and without wasm-udf, formatting, core Clippy and a Python API
snapshot comparison covering 108 exported classes. Workspace Clippy is limited
by the missing wasm target; live-server parity remains outstanding.
Generated-by: Isaac (OpenAI Codex, GPT-6).
> Share immutable field names across decoded rows in the Rust Spark Connect
> client
> --------------------------------------------------------------------------------
>
> Key: SPARK-60585
> URL: https://issues.apache.org/jira/browse/SPARK-60585
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Arrow row decoding repeatedly copies column names even though all rows in a
> record batch have the same names. Share immutable top-level names through
> Arc<[String]>, retaining owned names for locally constructed rows. Catalog
> and DataFrame use one common decoder. Nested Value::Struct names still copy
> per row and are outside this change.
> Implementation and benchmark table:
> https://github.com/apache/spark-connect-rust/pull/105
> Fresh comparison: Apache master a85e474f13b958fd3bd5ba7f91055dd2f2b959f3
> against PR head 6dccb5716d6bc52dd11beab061216ea034f7ad51. Independent
> screening and confirmation each measured 15 cases with 30 fresh processes per
> client/case: 2,700 successful samples total, zero failures. Source, release
> wheel/native, harness and policy identities were verified before and after
> measurement.
> Selected confirmation medians, baseline to candidate: 100,000 x 50 collect
> CPU 238.56 to 129.14 ms, wall 230.43 to 119.85 ms, held RSS 680.44 to 446.91
> MiB; 4,096 x 512 collect CPU 89.31 to 41.17 ms, wall 84.70 to 37.02 ms, held
> RSS 297.89 to 214.59 MiB. The fully scanned 4,096 x 512 case measured CPU
> 355.05 to 308.66 ms. RSS is process memory including allocator history.
> Canned Arrow/gRPC fixtures do not execute Spark plans.
> Overall CPU and memory acceptance remain inconclusive. All RSS regression
> guards passed in both runs, but timing intervals for expression/type
> construction, toArrow and catalog RPC guards remain uncertain under the
> unchanged policy. Further validation is required; selected case improvements
> do not override these guards.
> 399 Rust checks passed; all core integration targets compiled with and
> without wasm-udf. The 108-class Python API snapshot is unchanged. Formatting
> and core Clippy passed. Row remains 48 bytes, with the private RowFields
> representation 24 bytes on this 64-bit build. Local workspace Clippy lacks
> the wasm target. Green CI and live-server parity remain outstanding.
> Harness: https://github.com/dbtsai/pyspark-connect-benchmark
> Generated-by: Isaac (OpenAI Codex, GPT-6).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
