mingmwang commented on PR #5171: URL: https://github.com/apache/arrow-datafusion/pull/5171#issuecomment-1420458485
> I agree that fixing partitioning (global) and then sorting (local) is the more intuitive order, but this does not seem strictly necessary to me in theory. I can imagine changing global properties while still preserving the previous local properties for every partition (in the new plan). I think such a behavior would make rules very robust and easy to reason with. The current PR is not really about this anyway, but that's my general line of thinking when we refer to orthogonality. > > Nevertheless, maybe you are aware of a fundamental issue (that I am not foreseeing right now) which makes this impossible. Or, maybe doing this has some other negative consequences. If that turns out to be the case, then we will go with the current status quo, of course. If we implement the `RepartitionExec`/`CoalescePartitionsExec` with the sort preserving merge, we can make the two rules more orthogonally. But compared with the current random merge, sort preserving merge is not always a performance gain, especially in a distributed environment(like Ballista). We can discuss this in other tickets/issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
