alamb commented on issue #16353: URL: https://github.com/apache/datafusion/issues/16353#issuecomment-2963459115
> At the risk of making myself unpopular, I feel it's relevant to share my findings with you guys. Not at all --- this is great stuff -- thank you @pepijnve for continuing to push to get better. In my mind, given the tests that @zhuqi-lucas added in https://github.com/apache/datafusion/pull/16196, we are now in a great position to revisit the design. If we can come up with something simpler that still achieves the same aim that is a great outcome > The benefits I see of trying to leverage the same mechanism elsewhere in DataFusion are: > > * There is only one cooperative yielding mechanism at play. This is easier to reason about than multiple interacting ones. > * There is no need for additional API. DataFusion is already using this in the current released version. > * There are fewer corner cases. Once the budget is depleted, any point in the code checking the budget will yield since all those points are checking the same shared counter. A single framework that is general purpose and uses a mechanism in tokio certainly sounds compelling to me as well. I have somewhat lost track of what exactly you are proposing. Is it the approach in one of these PRs: - https://github.com/apache/datafusion/pull/16301 - https://github.com/apache/datafusion/pull/16319 Or is it something new based on some other research? What I think is important is that the solution 1. is well documented and clear 2. works with both built in operators as well as user defined ones 3. is minimally invasive (e.g. people implementing operators don't have to know too much about streams / polling, etc) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
