yongster opened a new issue, #10800:
URL: https://github.com/apache/arrow-rs/issues/10800
## Summary
`StringViewArray` / `BinaryViewArray` `filter`, `take`, and `interleave`
copy only the 16-byte views and then reattach the full `data_buffers()` list.
That is intentional zero-copy, and those kernels are fast because of it.
The cost shows up later: a 1% filter still holds 100% of the long-string
payload. Arrow IPC (file or network) then writes those unused variadic buffers
in full, so the message/file is billed at the unfiltered size.
`StringViewArray::gc()` already compact the buffers, and its docs say to use
it after filter/slice. `filter`/`take` do not call it. `BatchCoalescer` already
has a density check: copy only if actual data-buffer capacity is more than 2×
bytes actually referenced.
This is not "StringView is slower than Utf8". The compute kernels are
faster. The question is whether we should compact, and **where**, without
giving up that compute path by default.
## Local measurements
Local `--release` benchmark on current `arrow-rs` (Apple M4 Pro). Input:
50_000 rows × 200-byte `StringViewArray` (all long, so all bytes live in data
buffers). `filter`/`take` keep 1% (500 rows). Logical selected payload = 100
KiB.
| Result | Rows | Data buffer cap | IPC size |
|---|---|---|---|
| Filtered StringView (no gc) | 500 | 9.98 MiB (~105×) | 9.55 MiB |
| Same after `gc()` | 500 | 97.7 KiB | 106 KiB |
| Filtered Utf8 (control) | 500 | 97.7 KiB | 100 KiB |
Kernel medians: StringView `filter` 1.4 µs, Utf8 `filter` 7.3 µs, `gc()` of
the filtered view 3.9 µs.
Related: `concat` of two slices of the **same** view remounts the same
buffers again (400 rows, 80 KiB logical, ~19 MiB IPC). Parquet file size is
fine (it encodes selected values), but RSS while writing still holds the fat
buffers.
Root cause in `filter` (same pattern in `take`):
```rust
let buffers = array.data_buffers().to_vec();
unsafe { GenericByteViewArray::new_unchecked(views, buffers, nulls) }
```
## Position
**In-memory compute latency matters more than compacting every
intermediate.** Unconditionally calling `gc()` in `filter`/`take` would copy
even dense results (`gc()` always copies; the docs warn about that) and would
erase the zero-copy advantage when almost all rows are kept.
The default should stay zero-copy. Compaction should happen only when the
array is clearly sparse, or when we are about to serialize.
## Option A — compact by density threshold
After `filter`/`take` (maybe `concat`), call `gc()` only if:
```text
sum(data_buffers.capacity) > total_buffer_bytes_used * N
```
`N = 2` is already used as `need_gc` in
`arrow-select/src/coalesce/byte_view.rs`.
*Pros:* no caller changes; sparse pipelines get smaller RSS and IPC;
consistent with coalescing.
*Cons:* sparse `filter` pays an extra copy (~3.9 µs in the 1% case above,
still often cheaper than Utf8); the threshold will be bikeshed; pipelines that
filter and immediately drop the batch pay for a copy they do not need.
## Option B — compact at the IPC / serialization boundary
Keep `filter`/`take` zero-copy. Compact unused view buffers in
`IpcWriteOptions` when encoding a batch (file, stream, Flight). Compute code
does not change; we only copy when turning an Array into bytes.
*Pros:* compute stays fastest by default; IPC is the place that currently
writes unused buffers verbatim, so that is a precise time to pay.
*Cons:* in-memory RSS stays fat until something serializes; later `concat`
can still remount the same buffers; shared-memory / FFI consumers that never go
through IPC would not benefit.
A per-`filter` flag is intentionally not proposed. Most callers would not
know to set it, and it would push a serialization concern into the compute API.
## Questions
1. Is this a compute-kernel concern, a serialization-boundary concern, or
both?
2. If a threshold: reuse the coalescer's 2×? Make it configurable?
3. If Option B: should `IpcWriteOptions::compact_view_buffers` default on or
off?
4. Should remounting the same `Buffer` in `concat` (two slices of one view)
be this issue or a follow-up?
5. `ListView` `filter`/`take` leaving the full child is the same design.
Same ticket or separate?
## Preference for discussion (not a decision)
Keep `filter`/`take` zero-copy by default.
Prefer either or both of:
- compact view buffers when writing IPC (Option B);
- auto-`gc()` only when clearly sparse (Option A, reuse the 2× coalescer
rule).
Do **not** unconditionally `gc()` on every `filter`/`take`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
