adriangb opened a new pull request, #12185: URL: https://github.com/apache/arrow-rs/pull/12185
# Which issue does this PR close? - Closes #12182. # Rationale for this change The IPC writer writes the full child values of a sliced `ListView` or `LargeListView`. A 10-row slice of a large list view costs about as much as the whole parent. `List`, `LargeList`, `Map` and sparse `Union` are already narrowed to the slice. # What changes are included in this PR? `get_list_view_array_buffers` narrows the child to the values that the slice references and rebases the offsets. Views can overlap or be out of order, so the range runs from the smallest offset to the largest `offset + size` of the non-empty views. Empty and null views with size 0 get offset 0, because their offsets can lie outside that range. Values inside the range that no view references are still written. Stream sizes for a 10-row slice, with 8-byte alignment: | Array | Whole | Slice before | Slice after | | --- | --- | --- | --- | | ListView<UInt32>, 100,000 rows | 2,012,792 | 1,200,368 | 608 | | ListView<Utf8>, 10,000 rows | 237,128 | 155,960 | 664 | | ListView<ListView<UInt32>>, 10,000 rows | 681,168 | 599,992 | 1,192 | # Are these changes tested? Yes. New tests write a whole array and a slice, check that both read back equal, and check that the slice is much smaller. They cover `ListView` and `LargeListView`, a list view of strings with nulls, nested list views, overlapping and out-of-order views, and slices that hold only empty or null views. Two tests also assert the exact rebased offsets and child range. # Are there any significant user-facing changes? No. Sliced list views produce smaller IPC output. # AI usage disclosure Claude Code wrote the fix and the tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
