Dandandan commented on PR #15380: URL: https://github.com/apache/datafusion/pull/15380#issuecomment-2803501353
Hm that doesn't make much sense as > > Thanks for sharing the results @zhuqi-lucas this is really interesting! > > I think it mainly shows that we probably should try and use more efficient in memory sorting (e.g. an arrow kernel that sorts multiple batches) here rather than use `SortPreservingMergeStream` which is intended to be used on data streams. The arrow kernel would avoid the regressions of `concat`. > > I think the SortPreservingMergeStream is about as efficient as we know how to make it > > Maybe we can look into what overhead makes concat'ing better 🤔 Any per-stream overhead we can improve in SortPreservingMergeStream would likely flow directly to any query that does sorts Hm 🤔 ... but that will still take a separate step of sorting the input bathes, which next to sorting involves a full extra copy using `take` (slower than `concat`) followed by merging the batches. Also the built-in sort on the entire output is likely to be much faster than doing a merge on the outputs. I think the most efficient way would be to sort the indices to the arrays in one step followed by `interleave`, without either `concat` or `sort` followed by `merge` which would benefit the most from the built in sort algorithm and avoids copying the data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
