[
https://issues.apache.org/jira/browse/SPARK-60585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60585:
----------------------------
Description:
Share immutable field names across top-level Arrow-decoded rows and nested
struct values.
Implementation and benchmark table:
https://github.com/apache/spark-connect-rust/pull/105
Related work: SPARK-60585 and SPARK-60588; both remain open.
Apple M3 Max/macOS arm64; Apache master `a85e474` → combined PR head `834402b`
(top-level and nested sharing; no incremental nested-only comparison). The
[benchmark harness](https://github.com/dbtsai/pyspark-connect-benchmark) uses
canned Arrow/gRPC fixtures, 23 frozen cases, 30 fresh processes per case/client
and three rotating clients. screen: 2,070 recorded, 2,010 successful, 60
failed, 0 missing.
**Overall frozen verdict: rejected.** Arrow `Duration(Microsecond)` is
unsupported in both Rust builds (30 failed mixed-record samples each per run);
timing guards also remain uncertain. No accepted overall CPU or memory
improvement is claimed. Independent full confirmation is pending.
Rust `Value::Struct` now holds public `StructValue` with shared names and owned
values. Migrate construction to `Value::Struct(Row::new(fields,
values).into())` and use its field/value accessors. Tested Python Row behavior
is unchanged. Measured arm64 `Row` and `Value` both remain 48 bytes.
412 Rust checks passed; all core integration targets compiled with and without
`wasm-udf`. The 108-class Python API snapshot and 11 behavior/fixture
comparisons pass, as do formatting and core Clippy. Workspace Clippy lacks the
wasm standard library; live-server parity is unavailable and exact-head CI
remains pending.
Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
was:
Arrow row decoding repeatedly copies column names even though all rows in a
record batch have the same names. Share immutable top-level names through
Arc<[String]>, retaining owned names for locally constructed rows. Catalog and
DataFrame use one common decoder. Nested Value::Struct names still copy per row
and are outside this change.
Implementation and benchmark table:
https://github.com/apache/spark-connect-rust/pull/105
Fresh comparison: Apache master a85e474f13b958fd3bd5ba7f91055dd2f2b959f3
against measured PR head 6dccb5716d6bc52dd11beab061216ea034f7ad51; the later
commit changes tests only. Independent screening and confirmation each measured
15 cases with 30 fresh processes per client/case: 2,700 successful samples
total, zero failures. Source, release wheel/native, harness and policy
identities were verified before and after measurement.
Selected confirmation medians, baseline to candidate: 100,000 x 50 collect CPU
238.56 to 129.14 ms, wall 230.43 to 119.85 ms, held RSS 680.44 to 446.91 MiB;
4,096 x 512 collect CPU 89.31 to 41.17 ms, wall 84.70 to 37.02 ms, held RSS
297.89 to 214.59 MiB. The fully scanned 4,096 x 512 case measured CPU 355.05 to
308.66 ms. RSS is process memory including allocator history. Canned Arrow/gRPC
fixtures do not execute Spark plans.
Overall CPU and memory acceptance remain inconclusive. All RSS regression
guards passed in both runs, but timing intervals for expression/type
construction, toArrow and catalog RPC guards remain uncertain under the
unchanged policy. Further validation is required; selected case improvements do
not override these guards.
399 Rust checks passed; all core integration targets compiled with and without
wasm-udf. The 108-class Python API snapshot is unchanged. Formatting and core
Clippy passed. Row remains 48 bytes, with the private RowFields representation
24 bytes on this 64-bit build. Local workspace Clippy lacks the wasm target.
Green CI and live-server parity remain outstanding.
Harness: https://github.com/dbtsai/pyspark-connect-benchmark
Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
Nested struct field-name sharing follow-up:
https://issues.apache.org/jira/browse/SPARK-60588
> Share immutable field names across decoded rows in the Rust Spark Connect
> client
> --------------------------------------------------------------------------------
>
> Key: SPARK-60585
> URL: https://issues.apache.org/jira/browse/SPARK-60585
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Share immutable field names across top-level Arrow-decoded rows and nested
> struct values.
> Implementation and benchmark table:
> https://github.com/apache/spark-connect-rust/pull/105
> Related work: SPARK-60585 and SPARK-60588; both remain open.
> Apple M3 Max/macOS arm64; Apache master `a85e474` → combined PR head
> `834402b` (top-level and nested sharing; no incremental nested-only
> comparison). The [benchmark
> harness](https://github.com/dbtsai/pyspark-connect-benchmark) uses canned
> Arrow/gRPC fixtures, 23 frozen cases, 30 fresh processes per case/client and
> three rotating clients. screen: 2,070 recorded, 2,010 successful, 60 failed,
> 0 missing.
> **Overall frozen verdict: rejected.** Arrow `Duration(Microsecond)` is
> unsupported in both Rust builds (30 failed mixed-record samples each per
> run); timing guards also remain uncertain. No accepted overall CPU or memory
> improvement is claimed. Independent full confirmation is pending.
> Rust `Value::Struct` now holds public `StructValue` with shared names and
> owned values. Migrate construction to `Value::Struct(Row::new(fields,
> values).into())` and use its field/value accessors. Tested Python Row
> behavior is unchanged. Measured arm64 `Row` and `Value` both remain 48 bytes.
> 412 Rust checks passed; all core integration targets compiled with and
> without `wasm-udf`. The 108-class Python API snapshot and 11 behavior/fixture
> comparisons pass, as do formatting and core Clippy. Workspace Clippy lacks
> the wasm standard library; live-server parity is unavailable and exact-head
> CI remains pending.
> Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
