waynexia commented on issue #6782: URL: https://github.com/apache/arrow-datafusion/issues/6782#issuecomment-1805487141
Hi, I want to share a use case about ยง7.6 Query Rewrites / Optimizer Passes. Besides making domain-specific features, the optimizer interface is also an excellent place to make the plan "scale" across different compute nodes. I've been exploring how to run distributed queries over DataFusion. It is not something similar to Ballista, but an optimizer to "fuse" several engine instances across compute nodes ([RFC](https://github.com/GreptimeTeam/greptimedb/blob/develop/docs/rfcs/2023-05-09-distributed-planner.md)). At this stage it can push down some plans to the node which is closer to data, and then collect and merge the final result using the remaining part of plan. It's also implemented as an optimizer (an analyzer rule actually). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
