milenkovicm commented on PR #1372: URL: https://github.com/apache/datafusion-ballista/pull/1372#issuecomment-3922342451
> LGTM, thanks for building the skeleton and leaving the TODOs for future contributions. > > For better understanding, I've summarized my understanding of the AQE workflow below , @milenkovicm appreciate if you could check is it correctly reflect your design! Hopefully, this can helps subsequent reviewers. > > <img alt="Image" width="2000" height="752" src="https://private-user-images.githubusercontent.com/135958699/551692383-54dd9628-fe03-42dc-8847-277e2e05bddd.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzE0Mzc5MTgsIm5iZiI6MTc3MTQzNzYxOCwicGF0aCI6Ii8xMzU5NTg2OTkvNTUxNjkyMzgzLTU0ZGQ5NjI4LWZlMDMtNDJkYy04ODQ3LTI3N2UyZTA1YmRkZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjYwMjE4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI2MDIxOFQxODAwMThaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00Yzg3NTRjNWIwNWNmMjU5ZjVjNDIwZmIyYWE0MWUyMjQ2ZjFjNGIxNWVmOGVhMTIxNDJmZDExNGFiOWIwMmFjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.rP8qpRdNuKiQ7HLEiHzr3LYaUQ-C6lK0L0NCSkGWTWo";> yes, in high level it makes sense, thanks for sharing previous implementation will take a physical plan and make set of stages of it, this is good but in case a shortcut is detected we cant really change it as it is static. this proposal add a bit of flexibility, DAG can be changed, but on top of it stages could be changed, like changing partitioning or changing actual operator -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://private-user-images.githubusercontent.com/135958699/551692383-54dd9628-fe03-42dc-8847-277e2e05bddd.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzE0Mzc5MTgsIm5iZiI6MTc3MTQzNzYxOCwicGF0aCI6Ii8xMzU5NTg2OTkvNTUxNjkyMzgzLTU0ZGQ5NjI4LWZlMDMtNDJkYy04ODQ3LTI3N2UyZTA1YmRkZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjYwMjE4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI2MDIxOFQxODAwMThaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00Yzg3NTRjNWIwNWNmMjU5ZjVjNDIwZmIyYWE0MWUyMjQ2ZjFjNGIxNWVmOGVhMTIxNDJmZDExNGFiOWIwMmFjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.rP8qpRdNuKiQ7HLEiHzr3LYaUQ-C6lK0L0NCSkGWTWo"](https://private-user-images.githubusercontent.com/135958699/551692383-54dd9628-fe03-42dc-8847-277e2e05bddd.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzE0Mzc5MTgsIm5iZiI6MTc3MTQzNzYxOCwicGF0aCI6Ii8xMzU5NTg2OTkvNTUxNjkyMzgzLTU0ZGQ5NjI4LWZlMDMtNDJkYy04ODQ3LTI3N2UyZTA1YmRkZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjYwMjE4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI2MDIxOFQxODAwMThaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00Yzg3NTRjNWIwNWNmMjU5ZjVjNDIwZmIyYWE0MWUyMjQ2ZjFjNGIxNWVmOGVhMTIxNDJmZDExNGFiOWIwMmFjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.rP8qpRdNuKiQ7HLEiHzr3LYaUQ-C6lK0L0NCSkGWTWo"){kind=link}