[
https://issues.apache.org/jira/browse/SPARK-60585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60585:
----------------------------
Description:
Arrow row decoding repeatedly copies column names even though all rows in a
record batch have the same names. Share immutable top-level names through
Arc<[String]>, retaining owned names for locally constructed rows. Catalog and
DataFrame use one common decoder. Nested Value::Struct names still copy per row
and are outside this change.
Implementation and benchmark table:
https://github.com/apache/spark-connect-rust/pull/105
Fresh comparison: Apache master a85e474f13b958fd3bd5ba7f91055dd2f2b959f3
against measured PR head 6dccb5716d6bc52dd11beab061216ea034f7ad51; the later
commit changes tests only. Independent screening and confirmation each measured
15 cases with 30 fresh processes per client/case: 2,700 successful samples
total, zero failures. Source, release wheel/native, harness and policy
identities were verified before and after measurement.
Selected confirmation medians, baseline to candidate: 100,000 x 50 collect CPU
238.56 to 129.14 ms, wall 230.43 to 119.85 ms, held RSS 680.44 to 446.91 MiB;
4,096 x 512 collect CPU 89.31 to 41.17 ms, wall 84.70 to 37.02 ms, held RSS
297.89 to 214.59 MiB. The fully scanned 4,096 x 512 case measured CPU 355.05 to
308.66 ms. RSS is process memory including allocator history. Canned Arrow/gRPC
fixtures do not execute Spark plans.
Overall CPU and memory acceptance remain inconclusive. All RSS regression
guards passed in both runs, but timing intervals for expression/type
construction, toArrow and catalog RPC guards remain uncertain under the
unchanged policy. Further validation is required; selected case improvements do
not override these guards.
399 Rust checks passed; all core integration targets compiled with and without
wasm-udf. The 108-class Python API snapshot is unchanged. Formatting and core
Clippy passed. Row remains 48 bytes, with the private RowFields representation
24 bytes on this 64-bit build. Local workspace Clippy lacks the wasm target.
Green CI and live-server parity remain outstanding.
Harness: https://github.com/dbtsai/pyspark-connect-benchmark
Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
Nested struct field-name sharing follow-up:
https://issues.apache.org/jira/browse/SPARK-60588
was:
Arrow row decoding repeatedly copies column names even though all rows in a
record batch have the same names. Share immutable top-level names through
Arc<[String]>, retaining owned names for locally constructed rows. Catalog and
DataFrame use one common decoder. Nested Value::Struct names still copy per row
and are outside this change.
Implementation and benchmark table:
https://github.com/apache/spark-connect-rust/pull/105
Fresh comparison: Apache master a85e474f13b958fd3bd5ba7f91055dd2f2b959f3
against PR head 6dccb5716d6bc52dd11beab061216ea034f7ad51. Independent screening
and confirmation each measured 15 cases with 30 fresh processes per
client/case: 2,700 successful samples total, zero failures. Source, release
wheel/native, harness and policy identities were verified before and after
measurement.
Selected confirmation medians, baseline to candidate: 100,000 x 50 collect CPU
238.56 to 129.14 ms, wall 230.43 to 119.85 ms, held RSS 680.44 to 446.91 MiB;
4,096 x 512 collect CPU 89.31 to 41.17 ms, wall 84.70 to 37.02 ms, held RSS
297.89 to 214.59 MiB. The fully scanned 4,096 x 512 case measured CPU 355.05 to
308.66 ms. RSS is process memory including allocator history. Canned Arrow/gRPC
fixtures do not execute Spark plans.
Overall CPU and memory acceptance remain inconclusive. All RSS regression
guards passed in both runs, but timing intervals for expression/type
construction, toArrow and catalog RPC guards remain uncertain under the
unchanged policy. Further validation is required; selected case improvements do
not override these guards.
399 Rust checks passed; all core integration targets compiled with and without
wasm-udf. The 108-class Python API snapshot is unchanged. Formatting and core
Clippy passed. Row remains 48 bytes, with the private RowFields representation
24 bytes on this 64-bit build. Local workspace Clippy lacks the wasm target.
Green CI and live-server parity remain outstanding.
Harness: https://github.com/dbtsai/pyspark-connect-benchmark
Generated-by: Isaac (OpenAI Codex, GPT-6).
> Share immutable field names across decoded rows in the Rust Spark Connect
> client
> --------------------------------------------------------------------------------
>
> Key: SPARK-60585
> URL: https://issues.apache.org/jira/browse/SPARK-60585
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Arrow row decoding repeatedly copies column names even though all rows in a
> record batch have the same names. Share immutable top-level names through
> Arc<[String]>, retaining owned names for locally constructed rows. Catalog
> and DataFrame use one common decoder. Nested Value::Struct names still copy
> per row and are outside this change.
> Implementation and benchmark table:
> https://github.com/apache/spark-connect-rust/pull/105
> Fresh comparison: Apache master a85e474f13b958fd3bd5ba7f91055dd2f2b959f3
> against measured PR head 6dccb5716d6bc52dd11beab061216ea034f7ad51; the later
> commit changes tests only. Independent screening and confirmation each
> measured 15 cases with 30 fresh processes per client/case: 2,700 successful
> samples total, zero failures. Source, release wheel/native, harness and
> policy identities were verified before and after measurement.
> Selected confirmation medians, baseline to candidate: 100,000 x 50 collect
> CPU 238.56 to 129.14 ms, wall 230.43 to 119.85 ms, held RSS 680.44 to 446.91
> MiB; 4,096 x 512 collect CPU 89.31 to 41.17 ms, wall 84.70 to 37.02 ms, held
> RSS 297.89 to 214.59 MiB. The fully scanned 4,096 x 512 case measured CPU
> 355.05 to 308.66 ms. RSS is process memory including allocator history.
> Canned Arrow/gRPC fixtures do not execute Spark plans.
> Overall CPU and memory acceptance remain inconclusive. All RSS regression
> guards passed in both runs, but timing intervals for expression/type
> construction, toArrow and catalog RPC guards remain uncertain under the
> unchanged policy. Further validation is required; selected case improvements
> do not override these guards.
> 399 Rust checks passed; all core integration targets compiled with and
> without wasm-udf. The 108-class Python API snapshot is unchanged. Formatting
> and core Clippy passed. Row remains 48 bytes, with the private RowFields
> representation 24 bytes on this 64-bit build. Local workspace Clippy lacks
> the wasm target. Green CI and live-server parity remain outstanding.
> Harness: https://github.com/dbtsai/pyspark-connect-benchmark
> Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
> Nested struct field-name sharing follow-up:
> https://issues.apache.org/jira/browse/SPARK-60588
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
