ferruzzi commented on code in PR #34324:
URL: https://github.com/apache/airflow/pull/34324#discussion_r1339180017
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks
*locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process
using one of the Local or Remote Executors. This is not correct. The executor
logic runs *inside* the scheduler process, and will run the tasks locally or
not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable
and any executor has access to all abilities and integrations within Airflow.
Primarily, the Airflow scheduler uses this interface to interact with the
executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the
:class:`~airflow.executors.base_executor.BaseExecutor`. You can look through
the code for the most detailed and up to date interface, but some important
highlights are outlined below:
+
+.. note::
+ For more information about Airflow's public interface see
:doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a
specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service but from your
preferred cloud provider.
+* You have a private tool/service for task execution that is only available to
you or your organization.
+
+
+Important BaseExecutor Methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These methods don't require overriding to implement your own executor, but are
useful to be aware of:
+
+* ``heartbeat``: The Airflow scheduler Job loop will periodically call
heartbeat on the executor. This is one of the main points of interaction
between the Airflow scheduler and the executor. This method updates some
metrics, triggers newly queued tasks to execute and updates state of
running/completed tasks.
+* ``queue_command``: The Airflow Executor will call this method of the
BaseExecutor to provide tasks to be run by the executor. The BaseExecutor
simply adds the TaskInstances to an internal list of queued tasks within the
executor.
+* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve
the current state of the TaskInstances the executor is executing.
+* ``has_task``: The scheduler uses this BaseExecutor method to determine if an
executor already has a specific task instance queued or running.
+* ``send_callback``: Sends any callbacks to the sink configured on the
executor.
+
+
+Mandatory Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods must be overridden at minimum to have your executor
supported by Airflow:
+
+* ``sync``: Sync will get called periodically during executor heartbeats.
Implement this method to update the state of the tasks which the executor knows
about. Optionally, attempting to execute queued tasks that have been received
from the scheduler.
+* ``execute_async``: executes a command asynchronously. A command in this
context is an Airflow CLI command to run an Airflow task. This method is called
(after a few layers) during executor heartbeat which is run periodically by the
scheduler. In practice, this method often just enqueues tasks into an internal
or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can
also execute the tasks directly as well (e.g. ``LocalExecutor``). This will
depend on the executor.
+
+
+Optional Interface Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods aren't required to override to have a functional Airflow
executor. However, some powerful capabilities and stability can come from
implementing them:
+
+* ``start``: The Airflow scheduler (and backfill) job will call this method
after it initializes the executor object. Any additional setup required by the
executor can be completed here.
+* end: The Airflow scheduler (and backfill) job will call this method as it is
tearing down. Any synchronous cleanup required to finish running jobs should be
done here.
Review Comment:
Vincent pointed this one out while I was typing. I'll resolve mine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
