adriangb opened a new pull request, #12039: URL: https://github.com/apache/arrow-rs/pull/12039
# Which issue does this PR close? - Related to #5513. No issue yet; this draft is to discuss the approach. # Rationale for this change When a slice of a `Utf8View` / `BinaryView` array is written to IPC, the writer writes every data buffer in full, so each slice carries the data of every row. Today callers must call `gc()` on each slice before writing it (as the comment in `write_array_data` suggests, and as DataFusion does before spilling). `gc()` copies the used bytes into new buffers, and then the writer copies them again into its output. This came up in https://github.com/apache/datafusion/pull/25910, which splits large spill batches into byte-bounded slices: every slice has to be compacted before it is written. The writer already narrows sliced primitive, byte, list, map and fixed-size list arrays. This PR adds an opt-in option that does the same for view data buffers, without the extra copy. # What changes are included in this PR? - `IpcWriteOptions::with_compact_view_buffers(bool)`, default `false`. - With it on, a view array is written as its views plus **one** data buffer holding the byte ranges the views use. Overlapping and adjacent ranges of a source buffer are written once, so views that share bytes (for example after `take`) keep sharing them. The `variadicBufferCounts` metadata records the one buffer. - Without compression, the ranges are written as zero-copy slices of the source buffers. To support a buffer written as several pieces, `EncodedBuffer` gets a `Segment` variant that carries its own padding. - With compression, the ranges are streamed into the LZ4 frame or Zstd encoder (`CompressionCodec::compress_segments_to_vec`), so they are not gathered into one buffer first. - Arrays whose data buffers hold more than `u32::MAX` bytes are written as before, since view offsets are `u32`. # Are these changes tested? Yes, new tests in `writer.rs`: - `compact_view_buffers_roundtrip_slices`: several data buffers, inline values, nulls, views that share bytes, `BinaryView`, and a view array inside a struct. It writes slices (including empty and one-row slices) with and without LZ4 and Zstd, through `StreamWriter`, `StreamEncoder` and `FileWriter`. It checks the round trip and that the output is smaller than without the option. - `compact_view_buffers_one_data_buffer_per_array`: the read array has one data buffer, holding exactly `total_buffer_bytes_used()` bytes. - `compact_view_buffers_inline_and_null_only`. Measurements from a throwaway harness (not in this PR) with a counting allocator. Each input is split into 16 slices and written to an `io::Write` that discards the data. The table shows bytes allocated to prepare and write the slices, comparing `gc()` on each slice then writing, against writing with this option. Bytes written and decoded sizes are equal, except where noted. | Input | Codec | `gc()` then write | `with_compact_view_buffers` | |---|---|---:|---:| | Utf8View, 512 × 64 KiB | none | 33.6 MB | 0.05 MB | | Utf8View, 512 × 64 KiB | LZ4 | 177.5 MB | 5.1 MB (file +5%) | | Utf8View, 512 × 64 KiB | Zstd | 67.3 MB | 0.6 MB | | Utf8View, 50k × 100 B | none | 5.8 MB | 2.4 MB | | Utf8View, 50k × 100 B | LZ4 | 150.8 MB | 105.6 MB | | Utf8View, 50k × 100 B | Zstd | 11.6 MB | 3.8 MB | Some of the LZ4 difference seems to come from the codec's own allocations for each buffer, not from the saved copy. I have not looked into that yet. # Are there any significant user-facing changes? A new opt-in `IpcWriteOptions` method. The default output is unchanged. Open questions: - Should dictionary values get the same treatment? They are written whole in a dictionary message today. - Is an option the right shape, or should this be the default for sliced view arrays? 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
