thinkharderdev commented on issue #11451: URL: https://github.com/apache/datafusion/issues/11451#issuecomment-2231287150
> Thanks, got it. And as I understand, it seems the reason why executors underutilized is when the cpu work is required, but the IO is in flight and not ready? > > So we try to spawn more io tasks to improve the io parallelism. However, the schedule work is carried out by `tokio`, so maybe it can't reach our goal about improving io parallelism? > > For example, assume we spawn: > > * 8 cpu tasks > * and 16 io tasks for improving io prallelism > > The ideal scenario is that: > > * 16 io tasks are scheduled and executed, fetched 16 batches, and start to fetch next 16, yield > * 8 cpu tasks are scheduled, consume the current 16 batches, yield > * 16 io tasks are scheduled, and the next 16 batches are ready now, get them, and start next 16, yield > * 8 cpu tasks are scheduled, consume, yield > ... > > However, the tokio may schedule the tasks not as our expected: > > * 8 io tasks 1 are scheduled and executed, fetched 8 batches, and start to fetch next 8, yield > * 8 cpu tasks are scheduled, consume the current 8 batches, yield > * 8 io tasks 1 are scheduled and not ready. > * 8 io task 2 are scheduled and not ready, yet. > ... Yeah, so my thinking here is that the underlying issue is that if you have CPU work that is not yielding back to the scheduler enough it will stall the tokio reactor and in-flight IO work will not be able to make progress. So by forcing CPU-heavy tasks to yield back to the scheduler more then it will be more likely that IO tasks can make progress even when CPU-heavy tasks are consuming a lot of resources. Latencies are always going to suffer if you have more parallel CPU work than cores to run it on, but ideally by forcing the CPU-heavy tasks to yield back to the scheduler more we can reduce tail latencies so you merely get a linear increase in tail latencies if you oversubscribe CPU-heavy work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
