sunchao opened a new pull request, #11279: URL: https://github.com/apache/arrow-rs/pull/11279
# Which issue does this PR close? Part of #7184; related to #7186 and #11174. # Rationale for this change Selecting a few StringView or BinaryView rows can keep a large input buffer alive. A spilling operator may need those selected rows in separate owned storage so it can release the original batch. This proposes an explicit compact interleave operation. It copies selected valid values directly into fresh buffers, avoiding the intermediate array and source-buffer remapping built by `interleave` followed by `gc()`. Callers decide when copying is appropriate; the existing interleave path keeps its sharing behavior. # What changes are included in this PR? Add `interleave_byte_view_compact<T: ByteViewType>` to `arrow_select::interleave`, supporting StringViewArray and BinaryViewArray. - Preserve selection order, duplicates and nulls, including sliced and multiple input arrays. - Copy only selected non-null payload; inline values require no data buffers. - Size the payload block from the selected bytes when it fits in one view block; larger totals use the builder's normal multi-buffer growth. - Document ownership, size errors and the fact that copying can increase total live memory while the inputs remain retained. The API is deliberately typed to byte views. Recursive minification and gather allocation estimates can be discussed separately under #7186; #11174 addresses the related coalescer interface. This draft requests feedback on the explicit API. # Are these changes tested? On the final source, with Rust 1.98.1: - `cargo test --locked -p arrow-select --all-features --features arrow-array/force_validate`: 445 library tests, 5 integration tests and 18 doc tests passed. - `cargo clippy --locked -p arrow-select --all-targets --all-features --features arrow-array/force_validate -- -D warnings`: passed. - Workspace formatting, changed-file spelling and diff whitespace checks passed. The new regressions cover multi-source sliced input, repeated rows, non-UTF8 binary values, the 12/13-byte inline boundary, nonempty payload hidden by nulls, empty/all-null selections, and independent ownership after dropping the sources. One case selects two 20-byte values from input containing more than 64 KiB of payload and verifies that the output payload is 40 bytes with less than 1 KiB retained capacity. Local Cargo execution used an isolated validation workspace with exact copies of the six relevant Arrow crates. Every resolved dependency version and checksum matches the upstream lockfile. Resolving the full workspace was blocked before compilation because the available registry mirror lacks the unrelated arrow-avro dependency `opendal 0.59.3`. Repository manifests and Cargo.lock are unchanged. Full CI is pending. # Are there any user-facing changes? A new opt-in function is available. Existing kernels and defaults are unchanged. This PR establishes explicit ownership behavior; it makes no measured throughput claim. AI assistance: Codex generated the implementation, tests and documentation. The diff received two independent source reviews in addition to the checks above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
