avantgardnerio commented on PR #25798: URL: https://github.com/apache/datafusion/pull/25798#issuecomment-6023806790
Having reviewed this more in depth, I think this is most useful within DataFusion itself. In Ballista we would make [BufferExec](https://github.com/apache/datafusion-ballista/pull/2095) implement this, and we'd have the executor stream driver release these when we did [unordered range repartitions](https://github.com/apache/datafusion-ballista/pull/2123). What would be more useful though would be a method on ExecutionPlan, defaulting to None, that existing pipeline breakers like SortExec override: ``` fn as_boundary(&self) -> Option<&dyn StageBoundary> ``` A driver could see if something could be treated as a stage boundary, and then optionally prime it as one without the need to wrap or double buffer. But the real killer use-case (and I'd love to see a demo) would be a [conditional release](https://github.com/apache/datafusion-ballista/blob/842ea906716723891972c204f2bec4c16d27205c/ballista/core/src/execution_plans/buffer.rs#L126-L138) if criteria were met. That would need is_ready to mean "safe to inspect" rather than only EOF. This would allow DataFusion (irrespective of dfd or Ballista) to do things such as dynamic build-side selection for joins. It could run both legs of a join up until some threshold, and then inspect them to see if either one completed under that threshold. If so it could use that leg as the build side and flip if necessary. I suspect this might help improve DF in real-world benchmarks and make the argument that this belongs in-repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
