kaxil commented on code in PR #40701:
URL: https://github.com/apache/airflow/pull/40701#discussion_r1679956398


##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -41,32 +49,175 @@ If you want to check which executor is currently set, you 
can use the ``airflow
     SequentialExecutor
 
 
+
 Executor Types
 --------------
 
 There are two types of executors - those that run tasks *locally* (inside the 
``scheduler`` process), and those that run their tasks *remotely* (usually via 
a pool of *workers*). Airflow comes configured with the ``SequentialExecutor`` 
by default, which is a local executor, and the simplest option for execution. 
However, the ``SequentialExecutor`` is not suitable for production since it 
does not allow for parallel task running and due to that, some Airflow features 
(e.g. running sensors) will not work properly. You should instead use the 
``LocalExecutor`` for small, single-machine production installations, or one of 
the remote executors for a multi-machine/cloud installation.
 
 
-**Local Executors**
+.. _executor-types-comparison:
+
+Local Executors
+^^^^^^^^^^^^^^^
+
+Airflow tasks are run locally within the scheduler process.
+
+**Pros**: Very easy to use, fast, very low latency, and few requirements for 
setup.
+
+**Cons**: Limited in capabilities and shares resources with the Airflow 
scheduler.
+
+**Examples**:
 
 .. toctree::
     :maxdepth: 1
 
     local
     sequential
 
-**Remote Executors**
+Remote Executors
+^^^^^^^^^^^^^^^^
+
+Remote executors can further be divided into two categories:
+
+*Queued/Batch Executors*
+
+Airflow tasks are sent to a central queue where remote workers pull tasks to 
execute. Often workers are persistent and run multiple tasks at once.
+
+**Pros**: More robust since you're decoupling workers from the scheduler 
process. Workers can be large hosts that can churn through many tasks (often in 
parallel) which is cost effective. Latency can be relatively low since workers 
can be provisioned to be running at all times to take tasks immediately from 
the queue.
+
+**Cons**: Shared workers have the noisy neighbor problem with tasks competing 
for resources on the shared hosts or competing for how the environment/system 
is configured. They can also be expensive if your workload is not constant, you 
may have workers idle, overly scaled in resources, or you have to manage 
scaling them up and down.
+
+**Examples**:
 
 * :doc:`CeleryExecutor <apache-airflow-providers-celery:celery_executor>`
-* :doc:`CeleryKubernetesExecutor 
<apache-airflow-providers-celery:celery_kubernetes_executor>`
-* :doc:`KubernetesExecutor 
<apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
-* :doc:`KubernetesLocalExecutor 
<apache-airflow-providers-cncf-kubernetes:local_kubernetes_executor>`
+* :doc:`BatchExecutor 
<apache-airflow-providers-amazon:executors/batch-executor>`
+
+
 
+*Containerized Executors*
+
+Airflow tasks are executed ad hoc inside containers/pods. Each task is 
isolated in its own containerized environment that is deployed when the Airflow 
task is queued.
+
+**Pros**: Each Airflow task is isolated to one container so no noisy neighbor 
problem. The execution environment can be customized for specific tasks (system 
libs, binaries, dependencies, amount of resources, etc). Cost effective as the 
workers are only alive for the duration of the task.
+
+**Cons**: There is latency on startup since the container or pod needs to 
deploy before the task can begin. Can be expensive if you're running many 
short/small tasks. No workers to manage however you must manage something like 
a Kubernetes cluster.
+
+**Examples**:
+
+* :doc:`KubernetesExecutor 
<apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
+* :doc:`EcsExecutor <apache-airflow-providers-amazon:executors/ecs-executor>`
 
 .. note::
 
     New Airflow users may assume they need to run a separate executor process 
using one of the Local or Remote Executors. This is not correct. The executor 
logic runs *inside* the scheduler process, and will run the tasks locally or 
not depending the executor selected.
 
+Using Multiple Executors Concurrently
+-------------------------------------
+
+.. warning::
+    Multiple executor configuration is an alpha/experimental feature at the 
moment and may be subject to change without warning.
+
+Starting with version 2.10.0, Airflow can now operate with a multi-executor 
configuration. Each executor has its own set of pros and cons, often they are 
trade-offs between latency, isolation and compute efficiency among other 
properties (see :ref:`here <executor-types-comparison>` for comparisons of 
executors). Running multiple executors allows you to make better use of the 
strengths of all the available executors and avoid their weaknesses. In other 
words, you can use a specific executor for a specific set of tasks where its 
particular merits and benefits make the most sense for that use case.
+
+Configuration
+^^^^^^^^^^^^^
+
+Configuring multiple executors uses the same configuration option (as 
described :ref:`here <executor-basic-configuration>`) as single executor use 
cases, leveraging a comma separated list notation to specify multiple executors.
+
+.. note::
+    The first executor in the list (either on its own or along with other 
executors) will behave the same as it did in pre-2.10.0 releases. In other 
words, this will be the default executor for the environment. Any Airflow Task 
or DAG that does not specify a specific executor will use this environment 
level executor. All other executors in the list will be initialized and ready 
to run tasks if specified on an Airflow Task or DAG. If you do not specify an 
executor in this configuration list, it cannot be used to run tasks.
+
+Some examples of valid multiple executor configuration:
+
+.. code-block:: ini
+
+    [core]
+    executor = 'LocalExecutor'
+
+.. code-block:: ini
+
+    [core]
+    executor = 'LocalExecutor,CeleryExecutor'
+
+.. code-block:: ini
+
+    [core]
+    executor = 'KubernetesExecutor,my.custom.module.ExecutorClass'
+
+
+.. note::
+    Using two instances of the _same_ executor class is not currently 
supported.
+
+To make it easier to specify executors on tasks and DAGs, executor 
configuration now supports aliases. You may then use this alias to refer to the 
executor in your DAGs (see below).
+
+.. code-block:: ini
+
+    [core]
+    executor = 'LocalExecutor,my.custom.module.ExecutorClass:ShortName'
+
+.. note::
+    If a DAG specifies a task to use an executor that is not configured, the 
DAG will fail to parse and a warning dialog will be shown in the Airflow UI. 
Please ensure that all executors you wish to use are specified in Airflow 
configuration on *any* host/container that is running an Airflow component 
(scheduler, workers, etc).
+
+Writing DAGs and Tasks
+^^^^^^^^^^^^^^^^^^^^^^
+
+To specify an executor for a task, make use of the executor parameter on 
Airflow Operators:
+
+.. code-block:: python
+
+    BashOperator(
+        task_id="hello_world",
+        executor="LocalExecutor",
+        bash_command="echo 'hello world!'",
+    )
+
+.. code-block:: python
+
+    @task(executor="LocalExecutor")
+    def hello_world():
+        print("hello world!")
+
+To specify an executor for an entire DAG, make use of the existing Airflow 
mechanism of default arguments. All tasks in the DAG will then use the 
specified executor (unless explicitly overridden by a specific task):
+
+.. code-block:: python
+
+    def hello_world():
+        print("hello world!")
+
+
+    def hello_world_again():
+        print("hello world again!")
+
+
+    with DAG(
+        dag_id="hello_worlds",
+        default_args={"executor": "LocalExecutor"},  # Applies to all tasks in 
the DAG
+    ) as dag:
+        # All tasks will use the executor from default args automatically
+        hw = hello_world()
+        hw_again = hello_world_again()
+
+.. note::
+    Tasks store the executor they were configured to run on in the Airflow 
database. Changes are reflected after each parsing of a DAG.
+
+Monitoring
+^^^^^^^^^^
+
+When using a single executor, Airflow metrics will behave as they were <2.9. 
But if multiple executors are configured then the executor metrics 
(``executor.open_slots``, ``executor.queued_slots``, and 
``executor.running_tasks``) will be published for each executor configured, 
with the executor name appended to the metric name (e.g. 
``executor.open_slots.<executor class name>``).

Review Comment:
   We should also cross-reference it in 
https://github.com/apache/airflow/blob/main/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
 since that is the only place folks might look at for metrics



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to