adriangb opened a new issue, #11458: URL: https://github.com/apache/arrow-rs/issues/11458
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** https://github.com/apache/arrow-rs/pull/11235 adds `PushBuffers::release_ranges`. The push decoder calls it each time it completes a row group, to release the bytes of that row group. `release_ranges` visits all buffered ranges on each call. A caller that buffers many ranges (for example, one that reads ahead for many row groups or pushes one buffer per page) pays `O(buffers)` for each completed row group, also when the release touches only a few buffers. **Describe the solution you'd like** Since https://github.com/apache/arrow-rs/pull/11237, the buffers are sorted by start and `max_len` is the length of the longest buffer. A buffer can overlap a released range `a..b` only if its start is in `a - max_len..b`. Thus, `release_ranges` can find the candidate buffers with a binary search, as `PushBuffers::find` does, and not visit the others. The trim and split logic does not change. Pushed buffers can overlap, and a split makes new parts that must go in sorted position. **Describe alternatives you've considered** Keep the linear scan. It is correct, and it is fast when the number of buffers is small. **Additional context** Follow-up to this review thread: https://github.com/apache/arrow-rs/pull/11235#discussion_r4212017992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
