sunchao commented on code in PR #11279:
URL: https://github.com/apache/arrow-rs/pull/11279#discussion_r4170834321
##########
arrow-select/src/interleave.rs:
##########
@@ -125,6 +128,89 @@ pub fn interleave(
}
}
+/// Interleaves byte view arrays into independently owned, compact buffers.
Review Comment:
I switched this to
`Interleaver::new().with_compact_byte_views(true).interleave(...)` and removed
the additional public free function. An optional
`with_preserve_byte_view_sharing(true)` copies a repeated source byte range
once into the output; it is separate because tracking ranges has a cost. These
options apply to top-level StringView/BinaryView arrays.
The existing free `interleave` retains its original dispatch and has no
executable reference to the compact kernels. The configurable method delegates
to it for defaults and other types. An integer-only client using the original
free API grew from 2,741,040 to 2,742,240 stripped bytes (+1,200, 0.044%) on
exact base `705e6405b` versus final head `d8726603c`, with Rust 1.99.0,
optimization level 3, one client codegen unit, and LTO off. Both clients
checked identical output. I kept the PR in draft while we settle the API; one
client does not establish a general downstream code-size guarantee.
##########
arrow-select/src/interleave.rs:
##########
@@ -125,6 +128,89 @@ pub fn interleave(
}
}
+/// Interleaves byte view arrays into independently owned, compact buffers.
+///
+/// Each pair in `indices` selects an array in `values` and a row in that
array,
+/// as in [`interleave`]. This supports both [`StringViewArray`] and
+/// [`BinaryViewArray`]. Only selected, non-null values are copied; inline
values
+/// need no data buffer. The result does not retain any input buffers.
+///
+/// Use this when selected rows must release their references to input storage,
+/// for example when partitioning a large batch for spilling. Unlike calling
+/// [`interleave`] followed by [`GenericByteViewArray::gc`], this does not
first
+/// construct an intermediate array referencing the input data buffers.
+///
+/// Copying can increase total memory usage while the inputs remain alive.
+/// Repeated selections copy their payload each time; values are not
deduplicated.
+/// Call [`interleave`] to share input data buffers instead.
+///
+/// # Errors
+///
+/// Returns an error if `values` is empty or the selected payload exceeds the
+/// supported size limits.
+///
+/// # Panics
+///
+/// Panics if an array or row index is out of bounds.
+///
+/// # Example
+///
+/// ```
+/// use arrow_array::StringViewArray;
+/// use arrow_select::interleave::interleave_byte_view_compact;
+///
+/// let a = StringViewArray::from(vec![Some("a long selected value"), None]);
+/// let b = StringViewArray::from(vec!["another selected value"]);
+/// let result = interleave_byte_view_compact(&[&a, &b], &[(1, 0), (0, 1), (0,
0)])?;
+/// assert_eq!(result, StringViewArray::from(vec![
+/// Some("another selected value"), None, Some("a long selected value")
+/// ]));
+/// # Ok::<(), arrow_schema::ArrowError>(())
+/// ```
+pub fn interleave_byte_view_compact<T: ByteViewType>(
Review Comment:
Yes, there is overlap, and #11174 is exploring interleaving in the
coalescer. The distinction I need here is an explicit ownership guarantee for
an arbitrary selection across multiple arrays: the returned top-level byte-view
array retains no input buffers. Today `push_batch_with_indices` takes rows from
one batch and then coalesces them; the byte-view coalescer can retain backing
buffers when its sparsity heuristic does not request a copy.
A coalescer interface with that explicit copying policy could serve this use
case too. I also checked `gc()`'s `copy_view_to_buffer`: it is a private method
in `arrow-array`, tied to a single source array and destination buffer. Reusing
it across crates would require exposing or extracting a cross-crate helper/API,
plus the selection, null handling, and optional range-sharing logic. This
revision keeps those changes local to interleave; whether to share that helper
or put the ownership option on the coalescer remains an API design question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
