2010YOUY01 commented on PR #25688: URL: https://github.com/apache/datafusion/pull/25688#issuecomment-6228184562
In https://github.com/apache/datafusion/pull/26360, I think our disagreement is whether more explicit invariants make maintenance easier. I think they do, while @alamb believes the opposite. Since I've dealt with fewer optimizer-related problems, I'll trust your judgment for now, though I'm not fully convinced yet. I'll perhaps look for more problem-specific, compelling invariant regions in the future. Regarding the next steps for the analyzer/optimizer split, I'd appreciate some clarification. As I understand it, this PR only separates the rules into phases, without actually enforcing the following invariant: ``` After `EnforceDistribution` (or after the analyzer phase), all subsequent rules must preserve distribution/ordering requirements, either by: - Recognizing `RepartitionExec` nodes and preserving them. - Reapplying `EnforceDistribution` internally if they break those requirements. ``` For now, this invariant is only documented as a guideline, with automatic validation potentially added in the future. If we want to enforce it, I think we need a machinery with similar complexity to https://github.com/apache/datafusion/pull/26360? If so, perhaps we could try to make it extensible to other invariant regions in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
