zhuqi-lucas commented on PR #16322: URL: https://github.com/apache/datafusion/pull/16322#issuecomment-2961608771
> > Especially in high-parallelism scenarios where polling all streams on every wake-up is unnecessarily expensive. > > @zhuqi-lucas my assumption is that in high-parallelism situations all those streams are distinct tasks with a channel on the end. I that incorrect? And is checking a channel for readiness such an expensive operation? > > What we're doing here is, as far as I understand it, a bespoke version of `tokio::select!` on 10k channels in the end. That should work, shouldn't it? @pepijnve Just saw the update for max 10k channels, i think it's harmless for performance, i agree we can also run sort-tpch to see the result. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
