Rich-T-kid commented on issue #7765: URL: https://github.com/apache/arrow-rs/issues/7765#issuecomment-5782560528
>maybe we could avoid any copies if all the slices originate from the same buffer 🤔 thought about this more and no we generally cant do this. after an aggregation `emit::All()` is called and then slices of arrays are passed to repartionExec. these all share the same backing array but they become non-contiguous because of hashing. we would need to re-allocate to new buffers in this case. the only case where we could just keep the backing buffer is when stretches of the array remain together but any chunk that isn't aligned ruins this. for example ``` backing buffer : [0-1000] slices : [0 - 200] [200 - 500] ``` this would be fine to re-use the backing buffer -> [0-500] but ``` slices : [0-200] [201-500] ``` is not okay to re-use. being able to avoid a O(N) copy can be huge though, I may be missing cases when we can avoid this work -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
