[
https://issues.apache.org/jira/browse/SPARK-60662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60662:
----------------------------
Fix Version/s: connect-rust-4.3.0
Description:
Current scope after the master rebase
The pandas row-conversion path clones the complete field-name vector into each
materialized row. Share one immutable field-name allocation across the rows of
a conversion using the existing StructValue and Row shared-name representation.
The remaining scope is pandas row conversion. The rewritten local-input path
already shares names, and supported columnar pandas fast paths can bypass row
materialization while retaining a fallback that still benefits from this change.
Preserve field-name and schema precedence, duplicate and Unicode names, owned
row values, null and empty input behavior, caller-mutation isolation, and
conversion/error behavior. Regression coverage should exercise the pandas path
and verify complete names and values while keeping ordinary local-input
semantics intact.
Rebenchmark the current head against current master before claiming CPU,
wall-time, or memory savings. Historical local tuple and Row results do not
validate this narrowed scope; fresh measurements should establish the remaining
pandas benefit. Live-Spark parity and current CI remain part of validation.
Proposed patch: https://github.com/apache/spark-connect-rust/pull/113
Historical pre-rebase description and validation
The text below records the original proposal and its historical measurements
and validation. It does not validate or measure the rebased patch. The current
scope above takes precedence.
The Python binding currently clones explicit-schema or pandas column names for
every row during local createDataFrame conversion. These names are immutable
within one conversion, so repeating their String allocations wastes conversion
work and retained memory.
Share the names once per conversion using the existing StructValue/Row
shared-name representation (Arc<[String]>). Preserve explicit-schema precedence
over Row/dict/pandas names, positional defaults, input parsing, owned values,
and error order. Each call must own its names so later caller mutations cannot
change a previously created DataFrame. No intended API or behavior change.
PR: https://github.com/apache/spark-connect-rust/pull/113
Validation should cover complete IPC names and values, duplicate/empty/Unicode
names, nulls, empty input, caller mutations, and overridden-input errors. The
PR reports 410 Rust checks and 274 offline Python tests passing, including four
isolated input-name tests. Full live Spark/official-suite parity remains
pending.
On macOS arm64/Python 3.12, independent confirmation medians against c36c846
show CPU reductions of 21.50% for 10,000 x 3 explicit Rows, 7.92% for 10,000 x
32 tuples, and 7.18% for 10,000 x 32 pandas input. The latter two held-RSS
reductions are 17.367 MiB and 22.461 MiB. Each screening/confirmation phase
used 30 fresh processes per build/case across 48 cases with a same-binary
control.
These are preliminary results: the draft has not passed the complete CPU or
memory acceptance policy. Five timing metrics per phase have inconclusive guard
confidence bounds; no peak-RSS target qualifies. Both Rust builds retain
pre-existing Duration(Microsecond) warmup failures tracked by SPARK-60607. RSS
does not establish allocation counts, and effects from separate patches are not
additive.
Frozen benchmark:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Related but distinct: SPARK-60585 and SPARK-60588 share decoded output
row/struct names; SPARK-60640 concerns intermediate Value clones during local
Arrow encoding.
was:
The Python binding currently clones explicit-schema or pandas column names for
every row during local createDataFrame conversion. These names are immutable
within one conversion, so repeating their String allocations wastes conversion
work and retained memory.
Share the names once per conversion using the existing StructValue/Row
shared-name representation (Arc<[String]>). Preserve explicit-schema precedence
over Row/dict/pandas names, positional defaults, input parsing, owned values,
and error order. Each call must own its names so later caller mutations cannot
change a previously created DataFrame. No intended API or behavior change.
PR: https://github.com/apache/spark-connect-rust/pull/113
Validation should cover complete IPC names and values, duplicate/empty/Unicode
names, nulls, empty input, caller mutations, and overridden-input errors. The
PR reports 410 Rust checks and 274 offline Python tests passing, including four
isolated input-name tests. Full live Spark/official-suite parity remains
pending.
On macOS arm64/Python 3.12, independent confirmation medians against c36c846
show CPU reductions of 21.50% for 10,000 x 3 explicit Rows, 7.92% for 10,000 x
32 tuples, and 7.18% for 10,000 x 32 pandas input. The latter two held-RSS
reductions are 17.367 MiB and 22.461 MiB. Each screening/confirmation phase
used 30 fresh processes per build/case across 48 cases with a same-binary
control.
These are preliminary results: the draft has not passed the complete CPU or
memory acceptance policy. Five timing metrics per phase have inconclusive guard
confidence bounds; no peak-RSS target qualifies. Both Rust builds retain
pre-existing Duration(Microsecond) warmup failures tracked by SPARK-60607. RSS
does not establish allocation counts, and effects from separate patches are not
additive.
Frozen benchmark:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Related but distinct: SPARK-60585 and SPARK-60588 share decoded output
row/struct names; SPARK-60640 concerns intermediate Value clones during local
Arrow encoding.
Summary: Share field names across pandas input rows in Spark Connect
Rust (was: Share field names across local input rows in Spark Connect Rust)
> Share field names across pandas input rows in Spark Connect Rust
> ----------------------------------------------------------------
>
> Key: SPARK-60662
> URL: https://issues.apache.org/jira/browse/SPARK-60662
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Current scope after the master rebase
> The pandas row-conversion path clones the complete field-name vector into
> each materialized row. Share one immutable field-name allocation across the
> rows of a conversion using the existing StructValue and Row shared-name
> representation.
> The remaining scope is pandas row conversion. The rewritten local-input path
> already shares names, and supported columnar pandas fast paths can bypass row
> materialization while retaining a fallback that still benefits from this
> change.
> Preserve field-name and schema precedence, duplicate and Unicode names, owned
> row values, null and empty input behavior, caller-mutation isolation, and
> conversion/error behavior. Regression coverage should exercise the pandas
> path and verify complete names and values while keeping ordinary local-input
> semantics intact.
> Rebenchmark the current head against current master before claiming CPU,
> wall-time, or memory savings. Historical local tuple and Row results do not
> validate this narrowed scope; fresh measurements should establish the
> remaining pandas benefit. Live-Spark parity and current CI remain part of
> validation.
> Proposed patch: https://github.com/apache/spark-connect-rust/pull/113
> Historical pre-rebase description and validation
> The text below records the original proposal and its historical measurements
> and validation. It does not validate or measure the rebased patch. The
> current scope above takes precedence.
> The Python binding currently clones explicit-schema or pandas column names
> for every row during local createDataFrame conversion. These names are
> immutable within one conversion, so repeating their String allocations wastes
> conversion work and retained memory.
> Share the names once per conversion using the existing StructValue/Row
> shared-name representation (Arc<[String]>). Preserve explicit-schema
> precedence over Row/dict/pandas names, positional defaults, input parsing,
> owned values, and error order. Each call must own its names so later caller
> mutations cannot change a previously created DataFrame. No intended API or
> behavior change.
> PR: https://github.com/apache/spark-connect-rust/pull/113
> Validation should cover complete IPC names and values,
> duplicate/empty/Unicode names, nulls, empty input, caller mutations, and
> overridden-input errors. The PR reports 410 Rust checks and 274 offline
> Python tests passing, including four isolated input-name tests. Full live
> Spark/official-suite parity remains pending.
> On macOS arm64/Python 3.12, independent confirmation medians against c36c846
> show CPU reductions of 21.50% for 10,000 x 3 explicit Rows, 7.92% for 10,000
> x 32 tuples, and 7.18% for 10,000 x 32 pandas input. The latter two held-RSS
> reductions are 17.367 MiB and 22.461 MiB. Each screening/confirmation phase
> used 30 fresh processes per build/case across 48 cases with a same-binary
> control.
> These are preliminary results: the draft has not passed the complete CPU or
> memory acceptance policy. Five timing metrics per phase have inconclusive
> guard confidence bounds; no peak-RSS target qualifies. Both Rust builds
> retain pre-existing Duration(Microsecond) warmup failures tracked by
> SPARK-60607. RSS does not establish allocation counts, and effects from
> separate patches are not additive.
> Frozen benchmark:
> https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
> Related but distinct: SPARK-60585 and SPARK-60588 share decoded output
> row/struct names; SPARK-60640 concerns intermediate Value clones during local
> Arrow encoding.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
