eladkal commented on code in PR #40701:
URL: https://github.com/apache/airflow/pull/40701#discussion_r1673001086


##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -41,32 +49,175 @@ If you want to check which executor is currently set, you 
can use the ``airflow
     SequentialExecutor
 
 
+
 Executor Types
 --------------
 
 There are two types of executors - those that run tasks *locally* (inside the 
``scheduler`` process), and those that run their tasks *remotely* (usually via 
a pool of *workers*). Airflow comes configured with the ``SequentialExecutor`` 
by default, which is a local executor, and the simplest option for execution. 
However, the ``SequentialExecutor`` is not suitable for production since it 
does not allow for parallel task running and due to that, some Airflow features 
(e.g. running sensors) will not work properly. You should instead use the 
``LocalExecutor`` for small, single-machine production installations, or one of 
the remote executors for a multi-machine/cloud installation.
 
 
-**Local Executors**
+.. _executor-types-comparison:
+
+Local Executors
+^^^^^^^^^^^^^^^
+
+Airflow tasks are run locally within the scheduler process.
+
+**Pros**: Very easy to use, fast, very low latency, and few requirements for 
setup.
+
+**Cons**: Limited in capabilities and shares resources with the Airflow 
scheduler.
+
+**Examples**:
 
 .. toctree::
     :maxdepth: 1
 
     local
     sequential
 
-**Remote Executors**
+Remote Executors
+^^^^^^^^^^^^^^^^
+
+Remote executors can further be divided into two categories:
+
+*Queued/Batch Executors*
+
+Airflow tasks are sent to a central queue where remote workers pull tasks to 
execute. Often workers are persistent and run multiple tasks at once.
+
+**Pros**: More robust since you're decoupling workers from the scheduler 
process. Workers can be large hosts that can churn through many tasks (often in 
parallel) which is cost effective. Latency can be relatively low since workers 
can be provisioned to be running at all times to take tasks immediately from 
the queue.
+
+**Cons**: Shared workers have the noisy neighbor problem with tasks competing 
for resources on the shared hosts or competing for how the environment/system 
is configured. They can also be expensive if your workload is not constant, you 
may have workers idle, overly scaled in resources, or you have to manage 
scaling them up and down.
+
+**Examples**:
 
 * :doc:`CeleryExecutor <apache-airflow-providers-celery:celery_executor>`
-* :doc:`CeleryKubernetesExecutor 
<apache-airflow-providers-celery:celery_kubernetes_executor>`
-* :doc:`KubernetesExecutor 
<apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
-* :doc:`KubernetesLocalExecutor 
<apache-airflow-providers-cncf-kubernetes:local_kubernetes_executor>`
+* :doc:`BatchExecutor 
<apache-airflow-providers-amazon:executors/batch-executor>`
+
+
 
+*Containerized Executors*
+
+Airflow tasks are executed ad hoc inside containers/pods. Each task is 
isolated in its own containerized environment that is deployed when the Airflow 
task is queued.
+
+**Pros**: Each Airflow task is isolated to one container so no noisy neighbor 
problem. The execution environment can be customized for specific tasks (system 
libs, binaries, dependencies, amount of resources, etc). Cost effective as the 
workers are only alive for the duration of the task.
+
+**Cons**: There is latency on startup since the container or pod needs to 
deploy before the task can begin. Can be expensive if you're running many 
short/small tasks. No workers to manage however you must manage something like 
a Kubernetes cluster.
+
+**Examples**:
+
+* :doc:`KubernetesExecutor 
<apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
+* :doc:`EcsExecutor <apache-airflow-providers-amazon:executors/ecs-executor>`
 
 .. note::
 
     New Airflow users may assume they need to run a separate executor process 
using one of the Local or Remote Executors. This is not correct. The executor 
logic runs *inside* the scheduler process, and will run the tasks locally or 
not depending the executor selected.
 
+Using Multiple Executors Concurrently
+-------------------------------------
+
+.. warning::
+    Multiple executor configuration is an alpha/experimental feature at the 
moment and may be subject to change without warning.
+
+Starting with version 2.10.0, Airflow can now operate with a multi-executor 
configuration. Each executor has its own set of pros and cons, often they are 
trade-offs between latency, isolation and compute efficiency among other 
properties (see :ref:`here <executor-types-comparison>` for comparisons of 
executors). Running multiple executors allows you to make better use of the 
strengths of all the available executors and avoid their weaknesses. In other 
words, you can use a specific executor for a specific set of tasks where its 
particular merits and benefits make the most sense for that use case.
+
+Configuration
+^^^^^^^^^^^^^
+
+Configuring multiple executors uses the same configuration option (as 
described :ref:`here <executor-basic-configuration>`) as single executor use 
cases, leveraging a comma separated list notation to specify multiple executors.
+
+.. note::
+    The first executor in the list (either on its own or along with other 
executors) will behave the same as it did in pre-2.10.0 releases. In other 
words, this will be the default executor for the environment. Any Airflow Task 
or DAG that does not specify a specific executor will use this environment 
level executor. All other executors in the list will be initialized and ready 
to run tasks if specified on an Airflow Task or DAG. If you do not specify an 
executor in this configuration list, it cannot be used to run tasks.

Review Comment:
   Should we say something about the previously mixed executors? 
`CeleryKubernetesExecutor` and others?
   
   I would imagine that some people use them and they might consider migrating 
to this feature. Do we need to warn about cases of
   
   `executor = 'CeleryKubernetesExecutor,EcsExecutor'`
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to