alamb commented on issue #24704: URL: https://github.com/apache/datafusion/issues/24704#issuecomment-5681440427
> Having nested Vec harm performance due to the extra indirection and add cost for the low cardinality case > instead we can implement it using mmap which complicate the implementation but will allow for better performance and less indirection since we will treat it as single flat vec. I don't fully understand what the mmap proposal is > instead what I did is supporting in arrow sort (I've coping the original arrow sort code to my branch) and allow to sort across multiple blocks while keeping them separated. This makes sense to me (basically a sort kernel that takes `Vec<Array>` and returns `Vec<Array>` where the arrays are reordered. We could also look into reusing the array allocations internally 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
