ozankabak commented on PR #4691: URL: https://github.com/apache/arrow-datafusion/pull/4691#issuecomment-1361522668
I am quite excited about this! It removes a lot of unnecessary pipeline-breaking sorts by (1) analyzing whether they are really necessary, and (2) transforming window queries to obviate the need for sorting. This PR not only optimizes Datafusion for existing use cases, but also propels Datafusion closer to being a great foundation for streaming use cases. It is a significant progress in the streaming roadmap we previously published. The PR looks big, but the meat of the change is mostly in a new file (the new rule). The rest of the changes are either quite small, or test-related. @mustafasrepo and I will be happy to answer any questions and are looking forward to feedback! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
