mobuchowski opened a new pull request #20443: URL: https://github.com/apache/airflow/pull/20443
This PR adds new Plugin API - listeners. It enables plugin authors to write [`pluggy` hook implementation](https://pluggy.readthedocs.io/en/stable/) that will be called on certain formalized extension points. To differentiate between current Airflow extension points, like plugins, and current Airflow hooks, implementations of those hooks are called `listeners`. The API is ment to be called across all dags, and all operators - in contrast to current `on_success_callback`, `pre_execute` and related family which are meant to provide callbacks for particular dag authors, or operator creators. `pluggy` mechanism enables us to execute multiple, or none, listeners that implement particular extension point, so that users can use multiple listeners seamlessly. In this PR, three such extension points are added. When TaskInstance's state is changed to `RUNNING`, `on_task_instance_running` hook is called. On change to`SUCCESS` `on_task_instance_success` is called, similarly on `FAILED` `on_task_instance_failed` is called. Actual notification mechanism is be implemented using [SQLAlchemy’s events mechanism](https://docs.sqlalchemy.org/en/13/orm/session_events.html#after-flush). This ensures that plugins will get every change of state, regardless of where in the codebase it happened. To make sure that this change is not affecting performance, running this mechanism on scheduler is disabled by default. The SQLAlchemy event mechanism is also not affected by default - the event listener is only added if we have any plugin which actually registers listener. Design doc: https://docs.google.com/document/d/1L3xfdlWVUrdnFXng1Di4nMQYQtzMfhvvWDR9K4wXnDU/edit?usp=sharing Background to this change: When discussing changing LineageBackend api for OpenLineage plugin, we came to consensus that the changes proposed there would be better suited to the new API https://github.com/apache/airflow/issues/17984 related: https://github.com/apache/airflow/issues/17984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
