berkaysynnada opened a new issue, #12300: URL: https://github.com/apache/datafusion/issues/12300
### Is your feature request related to a problem or challenge? During the initialization of cursors of SPM (SortPreservingMergeExec), it needs to get Poll::Ready() for each partition, to initiate the next partition cursor. We have experienced a scenario in our downstream where SPM is polling 1st partition continuously but never get a Poll::Ready(). Those polls get all upstream operator buffers and channels grow. However, if SPM have skipped that partition and tried to initiate other partitions before retry, we did not experience this problem (congestion in partition 1 is cleared once the other partitions are pulled). ### Describe the solution you'd like SPM does not wait initiating the next cursor partition. Instead, it iterates over all partitions continuously whether it is OK or not, until they are all successfully initiated. ### Describe alternatives you've considered _No response_ ### Additional context I have a test which actually needs to pass: https://github.com/synnada-ai/datafusion-upstream/tree/feature/spm-cursor-init It simply tries to collect results of a plan which consist of an SPM and a mock source exec. That mock source exec returns pending from the 1st partition until the 2nd partition is polled. Once the 2nd is polled, 1st one also starts to send its results. In the current version of `poll_next_inner()` of `SortPreservingMergeStream`, once the `maybe_poll_stream()` returns s a pending for the first partition, it never enters `poll_next_inner()` again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
