alamb commented on code in PR #6310:
URL: https://github.com/apache/arrow-datafusion/pull/6310#discussion_r1188816056
##########
datafusion/core/src/physical_plan/repartition/mod.rs:
##########
@@ -532,9 +541,28 @@ impl RepartitionExec {
timer.done();
}
- // If the input stream is endless, we may spin forever and never
yield back to tokio. Hence let us yield.
- // See https://github.com/apache/arrow-datafusion/issues/5278.
- tokio::task::yield_now().await;
+ // If the input stream is endless, we may spin forever and
Review Comment:
I think if the tokio executor has only a single thread and the input stream
can provide data infinitely, without a yield it will buffer the entire input
which seems non ideal
I agree https://github.com/apache/arrow-datafusion/issues/5278 as described
seems somewhat more like "when we used blocking IO with a single tokio thread
it blocked everything" -- as described on
https://github.com/apache/arrow-datafusion/issues/5278#issuecomment-1432901387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
