2010YOUY01 commented on PR #13423: URL: https://github.com/apache/datafusion/pull/13423#issuecomment-2480361560
The explanation for the problem is super clear One thing I don't understand is: For Tokio's convention, all tasks created by `spawn` are expected to be IO-bounded, and those CPU-bound task should be created using `spawn_blocking()`. I think the implementation also uses two separate thread pool, which is very similar to the approach in this doc. Why can't we all use `spawn_blocking()` for all CPU-bounded task, and instead we have to use two runtimes explicitly 🤔 If we follow this two runtime approach, would existing `spawn_blocking()` in the code (e.g. https://github.com/apache/datafusion/blob/6d8313ebc865f9bff007bfc04652f58b016cbc1b/datafusion/physical-plan/src/spill.rs#L53) cause unexpected behavior (3 thread pools co-exist) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
