peter-toth commented on PR #8817: URL: https://github.com/apache/arrow-datafusion/pull/8817#issuecomment-1902666804
As this PR (with `PlanContext` and `ExprContext`) takes some parts of the code into a certain direction, I think it would help review and would be useful for deciding on further improvements if you could document where do you think hidden composition requirements are. I'm referring to those that are not visible from the current upstram code but do exists in your downstream code or you think it might be useful in the future. To give more context @ozankabak wrote in https://github.com/apache/arrow-datafusion/issues/8913 that: > The main concern I have is composability. Downstream, we have many use cases where the flow goes something like this: > > 1. Visit/transform a given tree, and save per-node data during the process. > 2. Make a decision based depending on the final result of the visitation. > 3. Visit/transform the tree again, re-use the saved per-node data and only modify when necessary. > > (this can repeat multiple times) > Having worked with most of the physical optimization rules as well as some logical rules for a long time now (upstream and downstream), I think I can safely that this is necessary. As I mentioned before, it is also on our roadmap to contribute more physical optimizations upstream in the near future and they need this kind of composability/reusability. Do you think you can leave some comments to identidy the affected areas of the code? cc @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
