2010YOUY01 commented on issue #26225: URL: https://github.com/apache/datafusion/issues/26225#issuecomment-6143351883
Oh yes, thanks for pointing that out! This is exactly why I started thinking about these refactors. I want to add several features/improvements to HJ, NLJ, and Aggregate, but the existing implementations have become too convoluted to extend comfortably. So I took a step back and started rethinking the underlying architecture to figure out how to keep things organized. My origin targets are: - **HJ:** Spilling / existence-join fast path - **NLJ:** Better spilling / existence-join fast path / generic runtime-index framework #21921 I think the same approach could apply to other operators. If we have a major improvement idea but find the existing implementation complexity unmanageable, it might be worth revisiting the architecture before implementing the feature directly, as we've figured out a working pattern. One thing I'm particularly concerned about is pushing AI to add features without considering the added complexity. In the worst case, this could make the codebase increasingly buggy over time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
