kaxil commented on code in PR #40701:
URL: https://github.com/apache/airflow/pull/40701#discussion_r1679956398
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -41,32 +49,175 @@ If you want to check which executor is currently set, you
can use the ``airflow
SequentialExecutor
+
Executor Types
--------------
There are two types of executors - those that run tasks *locally* (inside the
``scheduler`` process), and those that run their tasks *remotely* (usually via
a pool of *workers*). Airflow comes configured with the ``SequentialExecutor``
by default, which is a local executor, and the simplest option for execution.
However, the ``SequentialExecutor`` is not suitable for production since it
does not allow for parallel task running and due to that, some Airflow features
(e.g. running sensors) will not work properly. You should instead use the
``LocalExecutor`` for small, single-machine production installations, or one of
the remote executors for a multi-machine/cloud installation.
-**Local Executors**
+.. _executor-types-comparison:
+
+Local Executors
+^^^^^^^^^^^^^^^
+
+Airflow tasks are run locally within the scheduler process.
+
+**Pros**: Very easy to use, fast, very low latency, and few requirements for
setup.
+
+**Cons**: Limited in capabilities and shares resources with the Airflow
scheduler.
+
+**Examples**:
.. toctree::
:maxdepth: 1
local
sequential
-**Remote Executors**
+Remote Executors
+^^^^^^^^^^^^^^^^
+
+Remote executors can further be divided into two categories:
+
+*Queued/Batch Executors*
+
+Airflow tasks are sent to a central queue where remote workers pull tasks to
execute. Often workers are persistent and run multiple tasks at once.
+
+**Pros**: More robust since you're decoupling workers from the scheduler
process. Workers can be large hosts that can churn through many tasks (often in
parallel) which is cost effective. Latency can be relatively low since workers
can be provisioned to be running at all times to take tasks immediately from
the queue.
+
+**Cons**: Shared workers have the noisy neighbor problem with tasks competing
for resources on the shared hosts or competing for how the environment/system
is configured. They can also be expensive if your workload is not constant, you
may have workers idle, overly scaled in resources, or you have to manage
scaling them up and down.
+
+**Examples**:
* :doc:`CeleryExecutor <apache-airflow-providers-celery:celery_executor>`
-* :doc:`CeleryKubernetesExecutor
<apache-airflow-providers-celery:celery_kubernetes_executor>`
-* :doc:`KubernetesExecutor
<apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
-* :doc:`KubernetesLocalExecutor
<apache-airflow-providers-cncf-kubernetes:local_kubernetes_executor>`
+* :doc:`BatchExecutor
<apache-airflow-providers-amazon:executors/batch-executor>`
+
+
+*Containerized Executors*
+
+Airflow tasks are executed ad hoc inside containers/pods. Each task is
isolated in its own containerized environment that is deployed when the Airflow
task is queued.
+
+**Pros**: Each Airflow task is isolated to one container so no noisy neighbor
problem. The execution environment can be customized for specific tasks (system
libs, binaries, dependencies, amount of resources, etc). Cost effective as the
workers are only alive for the duration of the task.
+
+**Cons**: There is latency on startup since the container or pod needs to
deploy before the task can begin. Can be expensive if you're running many
short/small tasks. No workers to manage however you must manage something like
a Kubernetes cluster.
+
+**Examples**:
+
+* :doc:`KubernetesExecutor
<apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
+* :doc:`EcsExecutor <apache-airflow-providers-amazon:executors/ecs-executor>`
.. note::
New Airflow users may assume they need to run a separate executor process
using one of the Local or Remote Executors. This is not correct. The executor
logic runs *inside* the scheduler process, and will run the tasks locally or
not depending the executor selected.
+Using Multiple Executors Concurrently
+-------------------------------------
+
+.. warning::
+ Multiple executor configuration is an alpha/experimental feature at the
moment and may be subject to change without warning.
+
+Starting with version 2.10.0, Airflow can now operate with a multi-executor
configuration. Each executor has its own set of pros and cons, often they are
trade-offs between latency, isolation and compute efficiency among other
properties (see :ref:`here <executor-types-comparison>` for comparisons of
executors). Running multiple executors allows you to make better use of the
strengths of all the available executors and avoid their weaknesses. In other
words, you can use a specific executor for a specific set of tasks where its
particular merits and benefits make the most sense for that use case.
+
+Configuration
+^^^^^^^^^^^^^
+
+Configuring multiple executors uses the same configuration option (as
described :ref:`here <executor-basic-configuration>`) as single executor use
cases, leveraging a comma separated list notation to specify multiple executors.
+
+.. note::
+ The first executor in the list (either on its own or along with other
executors) will behave the same as it did in pre-2.10.0 releases. In other
words, this will be the default executor for the environment. Any Airflow Task
or DAG that does not specify a specific executor will use this environment
level executor. All other executors in the list will be initialized and ready
to run tasks if specified on an Airflow Task or DAG. If you do not specify an
executor in this configuration list, it cannot be used to run tasks.
+
+Some examples of valid multiple executor configuration:
+
+.. code-block:: ini
+
+ [core]
+ executor = 'LocalExecutor'
+
+.. code-block:: ini
+
+ [core]
+ executor = 'LocalExecutor,CeleryExecutor'
+
+.. code-block:: ini
+
+ [core]
+ executor = 'KubernetesExecutor,my.custom.module.ExecutorClass'
+
+
+.. note::
+ Using two instances of the _same_ executor class is not currently
supported.
+
+To make it easier to specify executors on tasks and DAGs, executor
configuration now supports aliases. You may then use this alias to refer to the
executor in your DAGs (see below).
+
+.. code-block:: ini
+
+ [core]
+ executor = 'LocalExecutor,my.custom.module.ExecutorClass:ShortName'
+
+.. note::
+ If a DAG specifies a task to use an executor that is not configured, the
DAG will fail to parse and a warning dialog will be shown in the Airflow UI.
Please ensure that all executors you wish to use are specified in Airflow
configuration on *any* host/container that is running an Airflow component
(scheduler, workers, etc).
+
+Writing DAGs and Tasks
+^^^^^^^^^^^^^^^^^^^^^^
+
+To specify an executor for a task, make use of the executor parameter on
Airflow Operators:
+
+.. code-block:: python
+
+ BashOperator(
+ task_id="hello_world",
+ executor="LocalExecutor",
+ bash_command="echo 'hello world!'",
+ )
+
+.. code-block:: python
+
+ @task(executor="LocalExecutor")
+ def hello_world():
+ print("hello world!")
+
+To specify an executor for an entire DAG, make use of the existing Airflow
mechanism of default arguments. All tasks in the DAG will then use the
specified executor (unless explicitly overridden by a specific task):
+
+.. code-block:: python
+
+ def hello_world():
+ print("hello world!")
+
+
+ def hello_world_again():
+ print("hello world again!")
+
+
+ with DAG(
+ dag_id="hello_worlds",
+ default_args={"executor": "LocalExecutor"}, # Applies to all tasks in
the DAG
+ ) as dag:
+ # All tasks will use the executor from default args automatically
+ hw = hello_world()
+ hw_again = hello_world_again()
+
+.. note::
+ Tasks store the executor they were configured to run on in the Airflow
database. Changes are reflected after each parsing of a DAG.
+
+Monitoring
+^^^^^^^^^^
+
+When using a single executor, Airflow metrics will behave as they were <2.9.
But if multiple executors are configured then the executor metrics
(``executor.open_slots``, ``executor.queued_slots``, and
``executor.running_tasks``) will be published for each executor configured,
with the executor name appended to the metric name (e.g.
``executor.open_slots.<executor class name>``).
Review Comment:
We should also cross-reference it in
https://github.com/apache/airflow/blob/main/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
since that is the only place folks might look at for metrics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]