geoffreyclaude commented on issue #23174: URL: https://github.com/apache/datafusion/issues/23174#issuecomment-6192480028
We encountered the `SortExec` case described here in production: fragmented input creates one Tokio producer task per sort run, and the task buildup stalls unrelated queries for seconds. A local TPC-H workload reproduced this with a `SELECT 1` probe on the same runtime. #26242 bounds sort jobs across all partitions of one `SortExec`, capping on the runtime’s worker count. Each job acquires a permit **before spawning**, sorts one run, and exits. The merge then consumes the prepared streams. This addresses the existing unbounded spawning while leaving the broader internal-parallelism design open. The benchmark bot reported "no change" across all 11 SQL sort queries at both SF1 and SF10. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
