hugochinchilla commented on PR #28333: URL: https://github.com/apache/airflow/pull/28333#issuecomment-1352846394
@bolkedebruin what you propose escapes of the scope of a small incremental improvement, right now the scheduler decisions are evaluated in pure SQL and this MR adds a little more capabilities based on the same design. I think that what you propose is a very powerful idea but this can be added as a later improvement for a later release, I would like if we can avoid a "perfect is enemy of good" situation here. Also I have a concern with moving the evaluation of dependencies outside SQL, because of how the scheduling loop is executed it could happen that the query would return a set of rows, the rules would ignore them all, and do nothing, the next scheduling loop would return the same set of rows and the problem would repeat, making the scheduler starve while there are other DAGs available to execute. So this is not a change that can be made without rethinking a lot of assumptions on core parts that I'm not so familiar with. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
