dbtsai opened a new pull request, #105: URL: https://github.com/apache/spark-connect-rust/pull/105
### What changes were proposed in this pull request? Share immutable field names across rows decoded from each Arrow record batch. The decoder currently clones every field-name String for every row, even though all rows in a batch have the same names. A private `RowFields` representation keeps `Row::new` and `Row::empty` on their existing owned `Vec<String>` path. Decoded rows use `Arc<[String]>`. Field access, order, duplicate-name lookup, value ownership, content equality and Debug output remain unchanged. Row values still materialize as before. This draft contains only the row representation, Arrow row decoder and six focused tests. It is based on Apache master `a85e474`, after #102 landed, and adds only row-field sharing. ### Why are the changes needed? Wide collected results allocate and copy the same names repeatedly. Sharing the names once per batch removes that repeated work while keeping locally constructed rows on the owned path. #### Historical benchmark results — PR #102-based prototype The numbers below are screening measurements of the same owned/shared implementation on top of #102 at `ff6fddd3afce376895c802a5a0a113105060a1e8`, compared with that commit without the row change. **They are not measurements of this master-based PR head.** The current base includes the merged plan/expression sharing from #102; performance on this exact base still needs a fresh comparison. Harness: [pyspark-connect-benchmark](https://github.com/dbtsai/pyspark-connect-benchmark), frozen local revision `23bdf9046815ae615c921b2e6bab827eed00aaae`. Separate locked release wheels and Python environments; Apple M3 Max, macOS arm64, Python 3.12.14 and Rust 1.99.1. The 12-case screen used 15 fresh-process samples per client/case in rotating client order: **540 successful samples, zero failures** (360 Rust, 180 official Python reference). The table compares the two Rust builds; each cell is the median before -> after. | Workload | CPU ms | Elapsed ms | Held process RSS MiB | | --- | ---: | ---: | ---: | | 100,000 rows x 50 columns, collect | 232.173 -> 123.408 | 223.466 -> 115.189 | 680.05 -> 446.31 | | 4,096 rows x 512 columns, collect | 86.557 -> 39.881 | 82.213 -> 35.741 | 297.94 -> 214.53 | | 4,096 rows x 512 columns, collect and read every value | 383.609 -> 329.997 | 380.744 -> 326.667 | 300.44 -> 217.08 | | 100,000 rows x 8 columns, fully consumed iterator | 190.217 -> 165.350 | 185.864 -> 161.258 | 106.27 -> 106.75 | The fixture returns canned localhost gRPC/Arrow responses without a Spark execution engine. Imports, session setup, prepared inputs and validation are outside timing; return operations and full value scans are inside timing. Plain collect preserves lazy Python scalar conversion. RSS includes warmups and allocator history; it is steady-state process memory, not allocated bytes saved per call. External RSS sampling targets 2 ms and may miss brief peaks. The table is selected from the 12-case matrix; it is not full API coverage or live-server evidence. Wheel/native, source-patch and frozen harness/policy hashes were checked before and after the run. Raw artifacts and the local harness revision have not been published. **Overall benchmark verdict: inconclusive.** All RSS regression guards passed, but the following timing checks remain unresolved: - Expression construction CPU median: +16.09%, exploratory 95% interval -16.59% to +22.73%. Its wall-time interval is also too wide to establish safety. - Tuple-input CPU median: +2.86%, interval -1.15% to +5.64%, crossing the 5% regression allowance. These exploratory bootstrap intervals have no simultaneous multiple-case confidence guarantee. The owned/shared implementation has no independent confirmation run. An earlier all-shared representation was separately repeated, but those samples do not confirm this changed representation. Fresh master-based screening/confirmation, broader coverage and live-server parity remain outstanding; this draft does not claim an accepted overall speedup. ### Does this PR introduce _any_ user-facing change? No API or result-format change is intended. The representation is private, and the Rust and Python Row interfaces retain their existing behavior. The owned/shared enum can increase per-row metadata size; the input and return-path regression checks remain necessary. ### How was this patch tested? Fresh checks after rebasing this patch onto Apache master `a85e474`: - 398 core/unit, expression-sharing and selected protobuf golden tests passed, including six new tests for buffer ownership, shared names, clone independence, duplicate/order behavior, equality/Debug, empty rows, mismatched lengths and multi-batch decoding. - The release Python extension built; the exported 108-class member/signature snapshot is unchanged from the baseline release wheel. - Every core integration-test target compiled in release mode, both with and without `wasm-udf`. - Rust formatting, `git diff --check` and core-library Clippy passed; Clippy reports existing warnings. - Workspace Clippy is blocked because the local toolchain lacks `wasm32-unknown-unknown`, required by the `wasm-udfs` example. - Full official/live-server parity was not run: no local Connect server was listening on port 15002. This remains outstanding. The historical PR #102-based prototype separately passed 398 Rust checks; those are separate checks, not additional passes for this branch. ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Isaac (OpenAI Codex, GPT-6). This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
