[
https://issues.apache.org/jira/browse/SPARK-60666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60666:
----------------------------
Fix Version/s: connect-rust-4.3.0
Affects Version/s: connect-rust-4.3.0
(was: 5.0.0)
Description:
Current scope after the master rebase
Arrow-to-Row materialization currently repeats dynamic array downcasts and
container slicing while decoding values. Prepare a borrowed typed decoder once
per Arrow column and nested child, then reuse it during row materialization.
This is an Arrow result-decoding optimization, independent of local-input
inference and pandas-input routing, with no intended API or behavior change.
Returned Row and Value objects must continue to own their contents without
retaining the Arrow arrays after decoding.
Preserve row-major evaluation and error order, lazy errors for unused
unsupported children, absolute child offsets in sliced and nested arrays, null
handling, variant field selection, and exact scalar values. Regression coverage
should include nested lists, structs and maps; sliced arrays; empty batches and
containers; null parents; special floating-point bits; and conversion-error
ordering.
Rebenchmark the current head against current master before claiming CPU,
wall-time, or memory improvements. Historical benchmark figures do not validate
the rebased patch, and no current memory saving or allocation-count reduction
is established. Live-Spark parity and current CI remain part of validation.
Proposed patch: https://github.com/apache/spark-connect-rust/pull/112
Historical pre-rebase description and validation
The text below records the original proposal and its historical measurements
and validation. It does not validate or measure the rebased patch. The current
scope above takes precedence.
Arrow row materialization in spark-connect-rust repeatedly downcasts column
arrays and slices list/map children. Resolve a borrowed typed decoder once per
Arrow column and nested child, then reuse it during row materialization.
Preserve row-major evaluation, lazy errors, absolute child offsets, and owned
output values. No behavior or API change is intended. Add regression coverage
for sliced and nested arrays, nulls, exact scalar values, empty IPC boundary
batches, duplicate variant fields, and error ordering.
Proposed implementation: https://github.com/apache/spark-connect-rust/pull/112
The PR reports about 9% lower CPU and 8.6–8.9% lower wall time for fully
consumed depth-16 mixed output versus c36c846, with 0.83–0.94 MiB higher held
RSS. Confirmation used 30 processes/build/case across 38 cases, isolated
release wheels, and a same-binary control. No memory saving or allocation-count
reduction is established.
Benchmark harness:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Reported validation: 415 Rust unit/golden/sharing checks, both integration
compilation modes, formatting and core Clippy pass. A fresh release wheel
passes 273 offline Python tests, and the 108-class API snapshot is unchanged.
The PR remains draft: full live Spark/official-suite parity is pending.
Complete CPU/memory gates remain unmet due to known SPARK-60607 failures and
inconclusive regression bounds; workspace Clippy is blocked by a missing WASM
target.
was:
Arrow row materialization in spark-connect-rust repeatedly downcasts column
arrays and slices list/map children. Resolve a borrowed typed decoder once per
Arrow column and nested child, then reuse it during row materialization.
Preserve row-major evaluation, lazy errors, absolute child offsets, and owned
output values. No behavior or API change is intended. Add regression coverage
for sliced and nested arrays, nulls, exact scalar values, empty IPC boundary
batches, duplicate variant fields, and error ordering.
Proposed implementation: https://github.com/apache/spark-connect-rust/pull/112
The PR reports about 9% lower CPU and 8.6–8.9% lower wall time for fully
consumed depth-16 mixed output versus c36c846, with 0.83–0.94 MiB higher held
RSS. Confirmation used 30 processes/build/case across 38 cases, isolated
release wheels, and a same-binary control. No memory saving or allocation-count
reduction is established.
Benchmark harness:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Reported validation: 415 Rust unit/golden/sharing checks, both integration
compilation modes, formatting and core Clippy pass. A fresh release wheel
passes 273 offline Python tests, and the 108-class API snapshot is unchanged.
The PR remains draft: full live Spark/official-suite parity is pending.
Complete CPU/memory gates remain unmet due to known SPARK-60607 failures and
inconclusive regression bounds; workspace Clippy is blocked by a missing WASM
target.
Summary: Dispatch Arrow result decoding once per typed column in
Spark Connect Rust (was: Dispatch Arrow decoding once per typed column in
Spark Connect Rust)
> Dispatch Arrow result decoding once per typed column in Spark Connect Rust
> --------------------------------------------------------------------------
>
> Key: SPARK-60666
> URL: https://issues.apache.org/jira/browse/SPARK-60666
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Current scope after the master rebase
> Arrow-to-Row materialization currently repeats dynamic array downcasts and
> container slicing while decoding values. Prepare a borrowed typed decoder
> once per Arrow column and nested child, then reuse it during row
> materialization.
> This is an Arrow result-decoding optimization, independent of local-input
> inference and pandas-input routing, with no intended API or behavior change.
> Returned Row and Value objects must continue to own their contents without
> retaining the Arrow arrays after decoding.
> Preserve row-major evaluation and error order, lazy errors for unused
> unsupported children, absolute child offsets in sliced and nested arrays,
> null handling, variant field selection, and exact scalar values. Regression
> coverage should include nested lists, structs and maps; sliced arrays; empty
> batches and containers; null parents; special floating-point bits; and
> conversion-error ordering.
> Rebenchmark the current head against current master before claiming CPU,
> wall-time, or memory improvements. Historical benchmark figures do not
> validate the rebased patch, and no current memory saving or allocation-count
> reduction is established. Live-Spark parity and current CI remain part of
> validation.
> Proposed patch: https://github.com/apache/spark-connect-rust/pull/112
> Historical pre-rebase description and validation
> The text below records the original proposal and its historical measurements
> and validation. It does not validate or measure the rebased patch. The
> current scope above takes precedence.
> Arrow row materialization in spark-connect-rust repeatedly downcasts column
> arrays and slices list/map children. Resolve a borrowed typed decoder once
> per Arrow column and nested child, then reuse it during row materialization.
> Preserve row-major evaluation, lazy errors, absolute child offsets, and owned
> output values. No behavior or API change is intended. Add regression coverage
> for sliced and nested arrays, nulls, exact scalar values, empty IPC boundary
> batches, duplicate variant fields, and error ordering.
> Proposed implementation: https://github.com/apache/spark-connect-rust/pull/112
> The PR reports about 9% lower CPU and 8.6–8.9% lower wall time for fully
> consumed depth-16 mixed output versus c36c846, with 0.83–0.94 MiB higher held
> RSS. Confirmation used 30 processes/build/case across 38 cases, isolated
> release wheels, and a same-binary control. No memory saving or
> allocation-count reduction is established.
> Benchmark harness:
> https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
> Reported validation: 415 Rust unit/golden/sharing checks, both integration
> compilation modes, formatting and core Clippy pass. A fresh release wheel
> passes 273 offline Python tests, and the 108-class API snapshot is unchanged.
> The PR remains draft: full live Spark/official-suite parity is pending.
> Complete CPU/memory gates remain unmet due to known SPARK-60607 failures and
> inconclusive regression bounds; workspace Clippy is blocked by a missing WASM
> target.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
