pepijnve opened a new issue, #16321: URL: https://github.com/apache/datafusion/issues/16321
### Describe the bug When running a query like `select a from annotated_data_infinite2 order by b desc limit 10`, a `SortPreservingMergeStream` is created that merge sorts the presorted partitions. During the initial sorting phase of the query, the query execution will continuously hit https://github.com/apache/datafusion/blob/1daa5ed5cc51546904d45e23cc148601d973942a/datafusion/physical-plan/src/sorts/merge.rs#L236 where the stream wakes the current task for rescheduling. Since all the input streams will be pending for quite some time during the sort on a large data the execution ends up in a busy-waiting loop. ### To Reproduce - Add `print!("P"); stdout().flush();` as the first line of `SortPreservingMergeStream::poll_next_inner` - Run a sort on a large dataset with target partitions > 1 so that `SortPreservingMergeStream` is used. - Observe how often the `poll_next_inner` is hit ### Expected behavior - No busy-waiting during query execution ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
