mzabaluev commented on PR #11872: URL: https://github.com/apache/arrow-rs/pull/11872#issuecomment-6486321890
> One problem with my single-run slice suggestion: the slice keeps the whole child buffer alive, so selecting a small run still holds (and reports through `get_array_memory_size`) almost the whole input. A small run is cheap to copy anyway, so maybe slice only when the run covers at least half of the child (`2 * count >= values.len()`) and copy otherwise to avoid worst case spills/memory pressure? The CPU overhead of copying is proportional to the size of the run, and my benchmarking (not yet committed) shows that it's slower than slicing on runs that are e.g. 1/8 of the input. So there is a tradeoff, though this is still faster than the fallback in `main`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
