[
https://issues.apache.org/jira/browse/SPARK-60588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60588:
----------------------------
Description:
Nested struct field-name sharing is now included in PR #105 alongside
SPARK-60585; it is no longer deferred to a separate implementation.
Implementation and benchmark table:
https://github.com/apache/spark-connect-rust/pull/105
Related work: SPARK-60585 and SPARK-60588; both remain open.
Apple M3 Max/macOS arm64; Apache master `a85e474` → combined PR head `834402b`
(top-level and nested sharing; no incremental nested-only comparison). The
[benchmark harness](https://github.com/dbtsai/pyspark-connect-benchmark) uses
canned Arrow/gRPC fixtures, 23 frozen cases, 30 fresh processes per case/client
and three rotating clients. screen: 2,070 recorded, 2,010 successful, 60
failed, 0 missing.
**Overall frozen verdict: rejected.** Arrow `Duration(Microsecond)` is
unsupported in both Rust builds (30 failed mixed-record samples each per run);
timing guards also remain uncertain. No accepted overall CPU or memory
improvement is claimed. Independent full confirmation is pending.
Rust `Value::Struct` now holds public `StructValue` with shared names and owned
values. Migrate construction to `Value::Struct(Row::new(fields,
values).into())` and use its field/value accessors. Tested Python Row behavior
is unchanged. Measured arm64 `Row` and `Value` both remain 48 bytes.
412 Rust checks passed; all core integration targets compiled with and without
`wasm-udf`. The 108-class Python API snapshot and 11 behavior/fixture
comparisons pass, as do formatting and core Clippy. Workspace Clippy lacks the
wasm standard library; live-server parity is unavailable and exact-head CI
remains pending.
Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
was:
Arrow decoding still copies every nested struct field name for every row
because Value::Struct stores Vec<(String, Value)>. A wide struct column
therefore retains repeated name allocations even after PR #105 shares top-level
Row names.
Investigate sharing immutable nested struct names while preserving field order,
duplicate-name lookup, null handling, nested value ownership, equality/Debug
behavior and the Rust/Python interfaces. Keep locally constructed values and
Arrow-decoded values behaviorally equivalent.
Validate with independent correctness and round-trip tests plus repeated
fresh-process comparisons against the exact current base. Include wide struct
columns, nested arrays/maps of structs and fully consumed output scans; measure
CPU, wall time and process RSS, and guard unrelated inputs/return paths. Do not
infer nested savings from top-level row measurements.
Follow-up to SPARK-60585 and
https://github.com/apache/spark-connect-rust/pull/105 . This work is outside
that PR's scope.
Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
> Share nested struct field names in the Rust Spark Connect client
> ----------------------------------------------------------------
>
> Key: SPARK-60588
> URL: https://issues.apache.org/jira/browse/SPARK-60588
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Nested struct field-name sharing is now included in PR #105 alongside
> SPARK-60585; it is no longer deferred to a separate implementation.
> Implementation and benchmark table:
> https://github.com/apache/spark-connect-rust/pull/105
> Related work: SPARK-60585 and SPARK-60588; both remain open.
> Apple M3 Max/macOS arm64; Apache master `a85e474` → combined PR head
> `834402b` (top-level and nested sharing; no incremental nested-only
> comparison). The [benchmark
> harness](https://github.com/dbtsai/pyspark-connect-benchmark) uses canned
> Arrow/gRPC fixtures, 23 frozen cases, 30 fresh processes per case/client and
> three rotating clients. screen: 2,070 recorded, 2,010 successful, 60 failed,
> 0 missing.
> **Overall frozen verdict: rejected.** Arrow `Duration(Microsecond)` is
> unsupported in both Rust builds (30 failed mixed-record samples each per
> run); timing guards also remain uncertain. No accepted overall CPU or memory
> improvement is claimed. Independent full confirmation is pending.
> Rust `Value::Struct` now holds public `StructValue` with shared names and
> owned values. Migrate construction to `Value::Struct(Row::new(fields,
> values).into())` and use its field/value accessors. Tested Python Row
> behavior is unchanged. Measured arm64 `Row` and `Value` both remain 48 bytes.
> 412 Rust checks passed; all core integration targets compiled with and
> without `wasm-udf`. The 108-class Python API snapshot and 11 behavior/fixture
> comparisons pass, as do formatting and core Clippy. Workspace Clippy lacks
> the wasm standard library; live-server parity is unavailable and exact-head
> CI remains pending.
> Generated-by: OpenAI Codex CLI 0.163.0-alpha.10.2.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
