eladkal commented on code in PR #40701:
URL: https://github.com/apache/airflow/pull/40701#discussion_r1673001086
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -41,32 +49,175 @@ If you want to check which executor is currently set, you
can use the ``airflow
SequentialExecutor
+
Executor Types
--------------
There are two types of executors - those that run tasks *locally* (inside the
``scheduler`` process), and those that run their tasks *remotely* (usually via
a pool of *workers*). Airflow comes configured with the ``SequentialExecutor``
by default, which is a local executor, and the simplest option for execution.
However, the ``SequentialExecutor`` is not suitable for production since it
does not allow for parallel task running and due to that, some Airflow features
(e.g. running sensors) will not work properly. You should instead use the
``LocalExecutor`` for small, single-machine production installations, or one of
the remote executors for a multi-machine/cloud installation.
-**Local Executors**
+.. _executor-types-comparison:
+
+Local Executors
+^^^^^^^^^^^^^^^
+
+Airflow tasks are run locally within the scheduler process.
+
+**Pros**: Very easy to use, fast, very low latency, and few requirements for
setup.
+
+**Cons**: Limited in capabilities and shares resources with the Airflow
scheduler.
+
+**Examples**:
.. toctree::
:maxdepth: 1
local
sequential
-**Remote Executors**
+Remote Executors
+^^^^^^^^^^^^^^^^
+
+Remote executors can further be divided into two categories:
+
+*Queued/Batch Executors*
+
+Airflow tasks are sent to a central queue where remote workers pull tasks to
execute. Often workers are persistent and run multiple tasks at once.
+
+**Pros**: More robust since you're decoupling workers from the scheduler
process. Workers can be large hosts that can churn through many tasks (often in
parallel) which is cost effective. Latency can be relatively low since workers
can be provisioned to be running at all times to take tasks immediately from
the queue.
+
+**Cons**: Shared workers have the noisy neighbor problem with tasks competing
for resources on the shared hosts or competing for how the environment/system
is configured. They can also be expensive if your workload is not constant, you
may have workers idle, overly scaled in resources, or you have to manage
scaling them up and down.
+
+**Examples**:
* :doc:`CeleryExecutor <apache-airflow-providers-celery:celery_executor>`
-* :doc:`CeleryKubernetesExecutor
<apache-airflow-providers-celery:celery_kubernetes_executor>`
-* :doc:`KubernetesExecutor
<apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
-* :doc:`KubernetesLocalExecutor
<apache-airflow-providers-cncf-kubernetes:local_kubernetes_executor>`
+* :doc:`BatchExecutor
<apache-airflow-providers-amazon:executors/batch-executor>`
+
+
+*Containerized Executors*
+
+Airflow tasks are executed ad hoc inside containers/pods. Each task is
isolated in its own containerized environment that is deployed when the Airflow
task is queued.
+
+**Pros**: Each Airflow task is isolated to one container so no noisy neighbor
problem. The execution environment can be customized for specific tasks (system
libs, binaries, dependencies, amount of resources, etc). Cost effective as the
workers are only alive for the duration of the task.
+
+**Cons**: There is latency on startup since the container or pod needs to
deploy before the task can begin. Can be expensive if you're running many
short/small tasks. No workers to manage however you must manage something like
a Kubernetes cluster.
+
+**Examples**:
+
+* :doc:`KubernetesExecutor
<apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
+* :doc:`EcsExecutor <apache-airflow-providers-amazon:executors/ecs-executor>`
.. note::
New Airflow users may assume they need to run a separate executor process
using one of the Local or Remote Executors. This is not correct. The executor
logic runs *inside* the scheduler process, and will run the tasks locally or
not depending the executor selected.
+Using Multiple Executors Concurrently
+-------------------------------------
+
+.. warning::
+ Multiple executor configuration is an alpha/experimental feature at the
moment and may be subject to change without warning.
+
+Starting with version 2.10.0, Airflow can now operate with a multi-executor
configuration. Each executor has its own set of pros and cons, often they are
trade-offs between latency, isolation and compute efficiency among other
properties (see :ref:`here <executor-types-comparison>` for comparisons of
executors). Running multiple executors allows you to make better use of the
strengths of all the available executors and avoid their weaknesses. In other
words, you can use a specific executor for a specific set of tasks where its
particular merits and benefits make the most sense for that use case.
+
+Configuration
+^^^^^^^^^^^^^
+
+Configuring multiple executors uses the same configuration option (as
described :ref:`here <executor-basic-configuration>`) as single executor use
cases, leveraging a comma separated list notation to specify multiple executors.
+
+.. note::
+ The first executor in the list (either on its own or along with other
executors) will behave the same as it did in pre-2.10.0 releases. In other
words, this will be the default executor for the environment. Any Airflow Task
or DAG that does not specify a specific executor will use this environment
level executor. All other executors in the list will be initialized and ready
to run tasks if specified on an Airflow Task or DAG. If you do not specify an
executor in this configuration list, it cannot be used to run tasks.
Review Comment:
Should we say something about the preciously mixed executors?
`CeleryKubernetesExecutor` and others?
I would imagine that some people use them and they might consider migrating
to this feature. Do we need to warn about cases of
`executor = 'CeleryKubernetesExecutor,EcsExecutor'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]