This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new f2190274d4 Add compatibility note for Listeners (#39544)
f2190274d4 is described below
commit f2190274d44163e44d5a91bf8728278dce06dca2
Author: Jarek Potiuk <[email protected]>
AuthorDate: Fri May 10 15:17:20 2024 +0200
Add compatibility note for Listeners (#39544)
While running tests for older versions of Airflow, it turned out that
forward-compatibility is maintained for implemented interfaces but if
someone implements listener using the latest version of listener,
they will not be backwards-compatible with older versions of Airflow.
This PR adds a note about it to Airflow documentation.
---
.../administration-and-deployment/listeners.rst | 64 ++++++++++++++++++++++
1 file changed, 64 insertions(+)
diff --git a/docs/apache-airflow/administration-and-deployment/listeners.rst
b/docs/apache-airflow/administration-and-deployment/listeners.rst
index 655b7de010..550278e37f 100644
--- a/docs/apache-airflow/administration-and-deployment/listeners.rst
+++ b/docs/apache-airflow/administration-and-deployment/listeners.rst
@@ -110,3 +110,67 @@ Airflow defines the specification as `hookspec
<https://github.com/apache/airflo
To include the listener in your Airflow installation, include it as a part of
an :doc:`Airflow Plugin </authoring-and-scheduling/plugins>`.
Listener API is meant to be called across all DAGs and all operators. You
can't listen to events generated by specific DAGs. For that behavior, try
methods like ``on_success_callback`` and ``pre_execute``. These provide
callbacks for particular DAG authors or operator creators. The logs and
``print()`` calls will be handled as part of the listeners.
+
+
+Compatibility note
+------------------
+
+The listeners interface might change over time. We are using ``pluggy``
specifications which
+means that implementation of the listeners written for older versions of the
interface should be
+forward-compatible with future versions of Airflow.
+
+However, the opposite is not guaranteed, so if your listener is implemented
against a newer version of the
+interface, it might not work with older versions of Airflow. It is not a
problem if you target single version
+of Airflow, because you can adjust your implementation to the version of
Airflow you use, but it is important
+if you are writing plugins or extensions that could be used with different
versions of Airflow.
+
+For example if a new field is added to the interface (like the ``error`` field
in the
+``on_task_instance_failed`` method in 2.10.0), the listener implementation
will not handle the case when
+the field is not present in the event object and such listeners will only work
for Airflow 2.10.0 and later.
+
+In order to implement a listener that is compatible with multiple versions of
Airflow including using features
+and fields added in newer versions of Airflow, you should check version of
Airflow used and use newer version
+of the interface implementation, but for older versions of Airflow you should
use older version of the
+interface.
+
+For example if you want to implement a listener that uses the ``error`` field
in the
+``on_task_instance_failed``, you should use code like this:
+
+.. code-block:: python
+
+ from importlib.metadata import version
+ from packaging.version import Version
+ from airflow.listeners import hookimpl
+
+ airflow_version = Version(version("apache-airflow"))
+ if airflow_version >= Version("2.10.0"):
+
+ class ClassBasedListener:
+ ...
+
+ @hookimpl
+ def on_task_instance_failed(
+ self, previous_state, task_instance, error: None | str |
BaseException, session
+ ):
+ # Handle error case here
+ pass
+
+ else:
+
+ class ClassBasedListener: # type: ignore[no-redef]
+ ...
+
+ @hookimpl
+ def on_task_instance_failed(self, previous_state, task_instance,
session):
+ # Handle no error case here
+ pass
+
+
+List of changes in the listener interfaces since 2.8.0 when they were
introduced:
+
+
++-----------------+-----------------------------+---------------------------------------+
+| Airflow Version | Affected method | Change
|
++=================+=============================+=======================================+
+| 2.10.0 | ``on_task_instance_failed`` | An error field added to the
interface |
++-----------------+-----------------------------+---------------------------------------+
