This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch v2-2-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bc551c146efbaecb385357c664a905eb0a437d75 Author: john-jac <[email protected]> AuthorDate: Mon Oct 11 12:16:37 2021 -0700 Doc: Add Callbacks Section to Logging & Monitoring (#18842) Airflow users often ask for an easy way to monitor the success or failure of tasks and/or DAGs. The use of callbacks, such as on_success_callback, helps users monitor DAG runs in an easily configurable way. This document provides an overview of available callbacks, linked to the relevant section in the Tasks document. It also provides a simple example of calling a function on any task failure in the DAG, as well as a separate function that is called upon the success of the last task in the DAG. (cherry picked from commit eae7b2ebd9df1029a9975af817555b29ef5dc627) --- .../logging-monitoring/callbacks.rst | 81 ++++++++++++++++++++++ docs/apache-airflow/logging-monitoring/index.rst | 1 + 2 files changed, 82 insertions(+) diff --git a/docs/apache-airflow/logging-monitoring/callbacks.rst b/docs/apache-airflow/logging-monitoring/callbacks.rst new file mode 100644 index 0000000..77ac594 --- /dev/null +++ b/docs/apache-airflow/logging-monitoring/callbacks.rst @@ -0,0 +1,81 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + + +Callbacks +========= + +A valuable component of logging and monitoring is the use of task callbacks to act upon changes in state of a given task, or across all tasks in a given DAG. +For example, you may wish to alert when certain tasks have failed, or have the last task in your DAG invoke a callback when it succeeds. + +.. note:: + + Callback functions are only invoked when the task state changes due to execution by a worker. + As such, task changes set by the command line interface (:doc:`CLI <../usage-cli>`) or user interface (:doc:`UI <../ui>`) do not + execute callback functions. + +Callback Types +-------------- + +There are four types of task events that can trigger a callback: + +=========================================== ================================================================ +Name Description +=========================================== ================================================================ +``on_success_callback`` Invoked when the task :ref:`succeeds <concepts:task-instances>` +``on_failure_callback`` Invoked when the task :ref:`fails <concepts:task-instances>` +``sla_miss_callback`` Invoked when a task misses its defined :ref:`SLA <concepts:slas>` +``on_retry_callback`` Invoked when the task is :ref:`up for retry <concepts:task-instances>` +=========================================== ================================================================ + + +Example +------- + +In the following example, failures in any task call the ``task_failure_alert`` function, and success in the last task calls the ``dag_success_alert`` function: + +.. code-block:: python + + from datetime import datetime, timedelta + from airflow import DAG + from airflow.operators.dummy import DummyOperator + + + def task_failure_alert(context): + print(f"Task has failed, task_instance_key_str: {context['task_instance_key_str']}") + + + def dag_success_alert(context): + print(f"DAG has succeeded, run_id: {context['run_id']}") + + + with DAG( + dag_id="example_callback", + schedule_interval=None, + start_date=datetime(2021, 1, 1), + dagrun_timeout=timedelta(minutes=60), + catchup=False, + on_success_callback=None, + on_failure_callback=task_failure_alert, + tags=["example"], + ) as dag: + + task1 = DummyOperator(task_id="task1") + task2 = DummyOperator(task_id="task2") + task3 = DummyOperator(task_id="task3", on_success_callback=dag_success_alert) + task1 >> task2 >> task3 diff --git a/docs/apache-airflow/logging-monitoring/index.rst b/docs/apache-airflow/logging-monitoring/index.rst index 47c0d5f..58edec4 100644 --- a/docs/apache-airflow/logging-monitoring/index.rst +++ b/docs/apache-airflow/logging-monitoring/index.rst @@ -31,6 +31,7 @@ In addition to the standard logging and metrics capabilities, Airflow supports t logging-architecture logging-tasks metrics + callbacks check-health errors
