hugochinchilla commented on PR #28333: URL: https://github.com/apache/airflow/pull/28333#issuecomment-1357379006
I've cleaned up a bit the code. IMHO, the current state of datasets allow treating the dataset dependencies as an AND rule, with this change we have the OR rule. For more complex situations you can split your logic in multiple DAGs and combine the rules to create almost anything you need (as suggested by @potiuk here https://github.com/apache/airflow/discussions/28253#discussioncomment-4376780). Things like https://github.com/apache/airflow/pull/28455 are elevating the project complexity to a very high level just in case we want to do something else later, without even considering which are this potential use cases, and also before having anyone asking for it. I think to consider having something like a "rule engine" we should first define the use cases we want to support, and this should be backed by real needs, not hypothetical cases just to justify the existence of the added complexity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
