2010YOUY01 commented on issue #26481: URL: https://github.com/apache/datafusion/issues/26481#issuecomment-6377000196
> 3. Split `EnsureRequirements` internally into the insert-only core, exposed as a public function, and its performance passes. I agree that each transformation/tree walk should have a single responsibility. It might also be fine to have multiple tree walks per rule, but having one tree walk handle multiple different optimizations could make it harder to maintain (haven't check the details yet, but I think this is what you mean by splitting the rule? plz correct me if I'm wrong). > 5. Move the non-correctness rules from the analyzer to the optimizer one at a time, each made valid in, valid out. I think this should be decided case by case. Transformations in the analyzer zone are typically simpler (e.g., pattern matching doesn't need to account for `RepartitionExec`), so moving more rules to analyzer might even help maintenance. So the future direction I think is to establish the contract for analyzer, as long as the extensions follow that distribution/sorting contract, it's allowed to extend analyzer. > * Choosing `SymmetricHashJoinExec` for two unbounded inputs decides whether a query can run at all, so it belongs in the analyzer or in the planner. For this specific sub-rule, I think it should be split out, and don't need to be in the default optimizer pipeline. According to the discussion below, unbounded input is not the typical usage in datafusion, and multiplexing its special handling with default implementation harms maintenance. This seem to be another typical example, so a better way for them is to collect them into a new crate, if there is some common requirements by multiple streaming users (like an extension but maintained inside `datafusion` repo) - https://github.com/apache/datafusion/issues/24265 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
