andygrove opened a new issue, #2727: URL: https://github.com/apache/datafusion-ballista/issues/2727
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** When a submitted physical plan has a node that two consumers read, such as one shuffle feeding two joins, Ballista's planner turns each consumer's subtree into its own stages, so the shared producer runs once for each consumer. Comet Native Executors (apache/datafusion-comet#7288) submits whole Spark queries as physical plans, and Spark reuses exchanges (`ReusedExchangeExec`) and subqueries. **Describe the solution you'd like** Recognize a plan node shared by several consumers (the same `Arc`, or an equivalent node) and plan it as a single stage whose shuffle output every consumer reads. **Describe alternatives you've considered** The client could avoid sharing, but that loses Spark's exchange reuse. **Additional context** Noted in Comet's `native/native-executors/README.md` (apache/datafusion-comet#7289). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
