ashb opened a new pull request, #28558:
URL: https://github.com/apache/airflow/pull/28558
Previously it was only possible to set "policy" functions via
airflow_local_settings.py which is fine for "small clusters" but being
able to control some of these policies from installed
plugins/distributions is helpful in a few circumstances: it lets
"platforms" (either of the SaaS variety, or internal platform teams)
specify some common policies, but still let local Ariflow teams define
other policies using airflow_local_settings
This uses Pluggy[[1]] for plugin management (that we are already using
for the listener interface), and out of the box that
gives us the ability to have multiple implementations of the policy
functions where it makes sense
- on all the "mutation" hooks all implementations will be called. (i.e.
for the policies that don't return a value)
- for any policy function that returns a value (such as
`get_dagbag_import_timeout`) then the plugins will be tried in LIFO
order until a non-None value is returned. (No code needed for us --
Pluggy handles this out of the box with the `firstresult=True`
property on the hookspec decorator).
[1]: https://pluggy.readthedocs.io/en/latest/index.html
Todo:
- [ ] Document the way of adding pluggins (setuptools entrypoints) and mark
it experimental
- [ ] Mention that the argument names must match
- [ ] Issue deprecation warning in existing migration code
- [ ] ???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
