potiuk commented on issue #18402: URL: https://github.com/apache/airflow/issues/18402#issuecomment-924252379
This would be a backwards incompatible-change, and it is completely needed and one that could be easily might be overridden by default_args of the DAG - which I believe should be more than enough - you can easly set default execution_timeout at the DAG level, which I believe is the only reasonable "common" setting. I think it is a very bad idea to set execution timeout on "organisation" or "group" or "whole airflow instance" level. That would make an unexpected behaviours for people who would not know that rule - that's violates "explicit is better than implicit" Python rule. Even If you want to enforce it, then in your Task Policy you should not change value of the parameter but you should simply fail the Task if you see that it runs without the right parameter set. This is the same about #18323 (I am just going to comment on that) - you should FAIL the task rather than mutate it in this case if it violates the Policy. There are some cases where you would likely want to mutate the task, but I think vast majority of cases is to fail it when the Policy is not met. As opposed to "magic defaults" this has multiple good properties: * it is very explicit * it teaches people to use good practices (especially if your exceptions are informative, provide the user with explanation and guide them how to comply with the Policy. * it allows you to make some more fine-grained check (like check that the timeout is <1h for certain kinds of operators and < 2h for other kinds - and you have full control over that at Airflow instance level I am converting that issue into discussion - I would like to hear why you would not be able to use it this way., I see that as non-issue, more of a choice that the Airflow project made. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
