dheerajturaga commented on code in PR #49859: URL: https://github.com/apache/airflow/pull/49859#discussion_r2072164469
########## providers/edge3/docs/deployment.rst: ########## @@ -0,0 +1,172 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Edge Worker Deployment +====================== + +Edge Workers can be deployed outside of the central Airflow infrastructure. They +are connected to the Airflow API server via HTTP(s). The Edge Worker is a +lightweight component that can be deployed on any machine that has outbound +HTTP(s) access to the Airflow API server. The Edge Worker is designed to be +lightweight and easy to deploy. It allows you to run Airflow tasks on machines +that are not part of your main data center, e.g. edge servers. This also allows to +deploy only reduced dependencies on the edge worker. + +Here are a few imperative requirements for your workers: + +- ``airflow`` needs to be installed, and the Airflow CLI needs to be in the path. This includes + the Task SDK as well as the edge3 provider package. +- Airflow configuration settings should be homogeneous across the cluster and on the edge site +- Operators that are executed on the Edge Worker need to have their dependencies + met in that context. Please take a look to the respective provider package + documentations +- The worker needs to have access to the ``DAGS_FOLDER``, and you need to + synchronize the filesystems by your own means. A common setup would be to + store your ``DAGS_FOLDER`` in a Git repository and sync it across machines using + Chef, Puppet, Ansible, or whatever you use to configure machines in your + environment. If all your boxes have a common mount point, having your + pipelines files shared there should work as well + + +Minimum Airflow configuration settings for the Edge Worker to make it running is: + +- Section ``[core]`` + + - ``executor``: Executor must be set or added to be ``airflow.providers.edge3.executors.EdgeExecutor`` + - ``internal_api_secret_key``: An encryption key must be set on api-server and Edge Worker component as + shared secret to authenticate traffic. It should be a random string like the fernet key + (but preferably not the same). + +- Section ``[edge]`` + + - ``api_enabled``: Must be set to true. It is disabled intentionally to not expose + API endpoint by default. This is the endpoint the worker connects to. + In a future release a dedicated API server can be started. + - ``api_url``: Must be set to the URL which exposes the api endpoint as it is reachable from the + worker. Typically this looks like ``https://your-hostname-and-port/edge_worker/v1/rpcapi``. + +To kick off a worker, you need to setup Airflow and kick off the worker +subcommand + +.. code-block:: bash + + airflow edge worker + +Your worker should start picking up tasks as soon as they get fired in +its direction. To stop a worker running on a machine you can use: + +.. code-block:: bash + + airflow edge stop + +It will try to stop the worker gracefully by sending ``SIGINT`` signal to main +process as and wait until all running tasks are completed. Also in a console you can use +``Ctrl-C`` to stop the worker. + +If you want to monitor the remote activity and worker, use the UI plugin which +is included in the provider package and install it on the webserver and use the +"Admin" - "Edge Worker Hosts" and "Edge Worker Jobs" pages. +(Note: The plugin is not ported to Airflow 3.0 web UI at time of writing) + +If you want to check status of the worker via CLI you can use the command + +.. code-block:: bash + + airflow edge status + +Some caveats: + +- Tasks can consume resources. Make sure your worker has enough resources to run ``worker_concurrency`` tasks +- Make sure that the ``pool_slots`` of a Tasks matches with the ``worker_concurrency`` of the worker. + See also :ref:`edge_executor:concurrency_slots`. +- Queue names are limited to 256 characters + +See :doc:`apache-airflow:administration-and-deployment/modules_management` for details on how Python and Airflow manage modules. + +.. _deployment:maintenance: + +Worker Maintenance Mode +----------------------- + +Sometimes infrastructure needs to be maintained. The Edge Worker provides a +maintenance mode to +- Stop accepting new tasks +- Drain all ongoing work gracefully + +Also please note if the worker detects that the Airflow or Edge provider package version +is not the same as the one running on the API server, it will stop accepting new tasks and shut down gracefully. +This is to prevent running tasks with different versions of the code. + +Worker status can be checked via the web UI in the "Admin" - "Edge Worker Hosts" page. + +.. image:: img/worker_hosts.png + +.. note:: + + As of time of writing the web UI to see edge jobs and manage workers is not ported to Airflow 3.0. + Until this is available you can use the CLI commands as described in :ref:`deployment:maintenance-mgmt-cli`. + + +Worker maintenance can also be triggered via the CLI command on the machine that runs the worker. + +.. code-block:: bash + + airflow edge maintenance --comments "Some comments for the maintenance" on + +This will stop the local worker instance from accepting new tasks and will complete running tasks. +If you add the command argument ``--wait`` the CLI will wait until all +running tasks are completed before return. + +If you want to know the status of you local worker while waiting on maintenance you can +use the command + +.. code-block:: bash + + airflow edge status + +This will show the status of the local worker instance as JSON and the tasks running on it. + +The status and maintenance comments will also be shown in the web UI +in the "Admin" - "Edge Worker Hosts" page. + +.. image:: img/worker_maintenance.png + +The local worker instance can be started to fetch new tasks via the command + +.. code-block:: bash + + airflow edge maintenance off + +This will start the worker again and it will start accepting tasks again. + +.. _deployment:maintenance-mgmt-cli: + +Worker Maintenance Management CLI +--------------------------------- + +Besides the CLI command to trigger maintenance on the local worker instance, there is also a command to Review Comment: ```suggestion Besides the CLI command to trigger maintenance on the local worker instance, there are also additional commands to ``` ########## providers/edge3/docs/deployment.rst: ########## @@ -0,0 +1,172 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Edge Worker Deployment +====================== + +Edge Workers can be deployed outside of the central Airflow infrastructure. They +are connected to the Airflow API server via HTTP(s). The Edge Worker is a +lightweight component that can be deployed on any machine that has outbound +HTTP(s) access to the Airflow API server. The Edge Worker is designed to be +lightweight and easy to deploy. It allows you to run Airflow tasks on machines +that are not part of your main data center, e.g. edge servers. This also allows to +deploy only reduced dependencies on the edge worker. + +Here are a few imperative requirements for your workers: + +- ``airflow`` needs to be installed, and the Airflow CLI needs to be in the path. This includes + the Task SDK as well as the edge3 provider package. +- Airflow configuration settings should be homogeneous across the cluster and on the edge site +- Operators that are executed on the Edge Worker need to have their dependencies + met in that context. Please take a look to the respective provider package + documentations +- The worker needs to have access to the ``DAGS_FOLDER``, and you need to + synchronize the filesystems by your own means. A common setup would be to + store your ``DAGS_FOLDER`` in a Git repository and sync it across machines using + Chef, Puppet, Ansible, or whatever you use to configure machines in your + environment. If all your boxes have a common mount point, having your + pipelines files shared there should work as well + + +Minimum Airflow configuration settings for the Edge Worker to make it running is: + +- Section ``[core]`` + + - ``executor``: Executor must be set or added to be ``airflow.providers.edge3.executors.EdgeExecutor`` + - ``internal_api_secret_key``: An encryption key must be set on api-server and Edge Worker component as + shared secret to authenticate traffic. It should be a random string like the fernet key + (but preferably not the same). + +- Section ``[edge]`` + + - ``api_enabled``: Must be set to true. It is disabled intentionally to not expose + API endpoint by default. This is the endpoint the worker connects to. + In a future release a dedicated API server can be started. + - ``api_url``: Must be set to the URL which exposes the api endpoint as it is reachable from the + worker. Typically this looks like ``https://your-hostname-and-port/edge_worker/v1/rpcapi``. + +To kick off a worker, you need to setup Airflow and kick off the worker +subcommand + +.. code-block:: bash + + airflow edge worker + +Your worker should start picking up tasks as soon as they get fired in +its direction. To stop a worker running on a machine you can use: + +.. code-block:: bash + + airflow edge stop + +It will try to stop the worker gracefully by sending ``SIGINT`` signal to main +process as and wait until all running tasks are completed. Also in a console you can use +``Ctrl-C`` to stop the worker. + +If you want to monitor the remote activity and worker, use the UI plugin which +is included in the provider package and install it on the webserver and use the +"Admin" - "Edge Worker Hosts" and "Edge Worker Jobs" pages. +(Note: The plugin is not ported to Airflow 3.0 web UI at time of writing) + +If you want to check status of the worker via CLI you can use the command + +.. code-block:: bash + + airflow edge status + +Some caveats: + +- Tasks can consume resources. Make sure your worker has enough resources to run ``worker_concurrency`` tasks +- Make sure that the ``pool_slots`` of a Tasks matches with the ``worker_concurrency`` of the worker. + See also :ref:`edge_executor:concurrency_slots`. +- Queue names are limited to 256 characters + +See :doc:`apache-airflow:administration-and-deployment/modules_management` for details on how Python and Airflow manage modules. + +.. _deployment:maintenance: + +Worker Maintenance Mode +----------------------- + +Sometimes infrastructure needs to be maintained. The Edge Worker provides a +maintenance mode to +- Stop accepting new tasks +- Drain all ongoing work gracefully + +Also please note if the worker detects that the Airflow or Edge provider package version +is not the same as the one running on the API server, it will stop accepting new tasks and shut down gracefully. +This is to prevent running tasks with different versions of the code. + +Worker status can be checked via the web UI in the "Admin" - "Edge Worker Hosts" page. + +.. image:: img/worker_hosts.png + +.. note:: + + As of time of writing the web UI to see edge jobs and manage workers is not ported to Airflow 3.0. + Until this is available you can use the CLI commands as described in :ref:`deployment:maintenance-mgmt-cli`. + + +Worker maintenance can also be triggered via the CLI command on the machine that runs the worker. + +.. code-block:: bash + + airflow edge maintenance --comments "Some comments for the maintenance" on + +This will stop the local worker instance from accepting new tasks and will complete running tasks. +If you add the command argument ``--wait`` the CLI will wait until all +running tasks are completed before return. + +If you want to know the status of you local worker while waiting on maintenance you can +use the command + +.. code-block:: bash + + airflow edge status + +This will show the status of the local worker instance as JSON and the tasks running on it. + +The status and maintenance comments will also be shown in the web UI +in the "Admin" - "Edge Worker Hosts" page. + +.. image:: img/worker_maintenance.png + +The local worker instance can be started to fetch new tasks via the command + +.. code-block:: bash + + airflow edge maintenance off + +This will start the worker again and it will start accepting tasks again. + +.. _deployment:maintenance-mgmt-cli: + +Worker Maintenance Management CLI +--------------------------------- + +Besides the CLI command to trigger maintenance on the local worker instance, there is also a command to +manage the maintenance of all workers in the cluster. This command can be used to trigger maintenance +on all workers in the cluster or to check the status of all workers in the cluster. + +This set commands need database access, such can only be called on the central Airflow Review Comment: ```suggestion These set of commands need database access, and can only be called on the central Airflow ``` ########## providers/edge3/docs/deployment.rst: ########## @@ -0,0 +1,172 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Edge Worker Deployment +====================== + +Edge Workers can be deployed outside of the central Airflow infrastructure. They +are connected to the Airflow API server via HTTP(s). The Edge Worker is a +lightweight component that can be deployed on any machine that has outbound +HTTP(s) access to the Airflow API server. The Edge Worker is designed to be +lightweight and easy to deploy. It allows you to run Airflow tasks on machines +that are not part of your main data center, e.g. edge servers. This also allows to +deploy only reduced dependencies on the edge worker. + +Here are a few imperative requirements for your workers: + +- ``airflow`` needs to be installed, and the Airflow CLI needs to be in the path. This includes + the Task SDK as well as the edge3 provider package. +- Airflow configuration settings should be homogeneous across the cluster and on the edge site +- Operators that are executed on the Edge Worker need to have their dependencies + met in that context. Please take a look to the respective provider package + documentations +- The worker needs to have access to the ``DAGS_FOLDER``, and you need to + synchronize the filesystems by your own means. A common setup would be to + store your ``DAGS_FOLDER`` in a Git repository and sync it across machines using + Chef, Puppet, Ansible, or whatever you use to configure machines in your + environment. If all your boxes have a common mount point, having your + pipelines files shared there should work as well + + +Minimum Airflow configuration settings for the Edge Worker to make it running is: + +- Section ``[core]`` + + - ``executor``: Executor must be set or added to be ``airflow.providers.edge3.executors.EdgeExecutor`` + - ``internal_api_secret_key``: An encryption key must be set on api-server and Edge Worker component as + shared secret to authenticate traffic. It should be a random string like the fernet key + (but preferably not the same). + +- Section ``[edge]`` + + - ``api_enabled``: Must be set to true. It is disabled intentionally to not expose + API endpoint by default. This is the endpoint the worker connects to. + In a future release a dedicated API server can be started. + - ``api_url``: Must be set to the URL which exposes the api endpoint as it is reachable from the + worker. Typically this looks like ``https://your-hostname-and-port/edge_worker/v1/rpcapi``. + +To kick off a worker, you need to setup Airflow and kick off the worker +subcommand + +.. code-block:: bash + + airflow edge worker + +Your worker should start picking up tasks as soon as they get fired in +its direction. To stop a worker running on a machine you can use: + +.. code-block:: bash + + airflow edge stop + +It will try to stop the worker gracefully by sending ``SIGINT`` signal to main +process as and wait until all running tasks are completed. Also in a console you can use +``Ctrl-C`` to stop the worker. + +If you want to monitor the remote activity and worker, use the UI plugin which +is included in the provider package and install it on the webserver and use the +"Admin" - "Edge Worker Hosts" and "Edge Worker Jobs" pages. +(Note: The plugin is not ported to Airflow 3.0 web UI at time of writing) + +If you want to check status of the worker via CLI you can use the command + +.. code-block:: bash + + airflow edge status + +Some caveats: + +- Tasks can consume resources. Make sure your worker has enough resources to run ``worker_concurrency`` tasks +- Make sure that the ``pool_slots`` of a Tasks matches with the ``worker_concurrency`` of the worker. + See also :ref:`edge_executor:concurrency_slots`. +- Queue names are limited to 256 characters + +See :doc:`apache-airflow:administration-and-deployment/modules_management` for details on how Python and Airflow manage modules. + +.. _deployment:maintenance: + +Worker Maintenance Mode +----------------------- + +Sometimes infrastructure needs to be maintained. The Edge Worker provides a +maintenance mode to +- Stop accepting new tasks +- Drain all ongoing work gracefully + +Also please note if the worker detects that the Airflow or Edge provider package version +is not the same as the one running on the API server, it will stop accepting new tasks and shut down gracefully. +This is to prevent running tasks with different versions of the code. + +Worker status can be checked via the web UI in the "Admin" - "Edge Worker Hosts" page. + +.. image:: img/worker_hosts.png + +.. note:: + + As of time of writing the web UI to see edge jobs and manage workers is not ported to Airflow 3.0. + Until this is available you can use the CLI commands as described in :ref:`deployment:maintenance-mgmt-cli`. + + +Worker maintenance can also be triggered via the CLI command on the machine that runs the worker. + +.. code-block:: bash + + airflow edge maintenance --comments "Some comments for the maintenance" on + +This will stop the local worker instance from accepting new tasks and will complete running tasks. +If you add the command argument ``--wait`` the CLI will wait until all +running tasks are completed before return. + +If you want to know the status of you local worker while waiting on maintenance you can +use the command + +.. code-block:: bash + + airflow edge status + +This will show the status of the local worker instance as JSON and the tasks running on it. + +The status and maintenance comments will also be shown in the web UI +in the "Admin" - "Edge Worker Hosts" page. + +.. image:: img/worker_maintenance.png + +The local worker instance can be started to fetch new tasks via the command + +.. code-block:: bash + + airflow edge maintenance off + +This will start the worker again and it will start accepting tasks again. + +.. _deployment:maintenance-mgmt-cli: + +Worker Maintenance Management CLI +--------------------------------- + +Besides the CLI command to trigger maintenance on the local worker instance, there is also a command to +manage the maintenance of all workers in the cluster. This command can be used to trigger maintenance Review Comment: ```suggestion manage the maintenance of all workers in the cluster. These commands can be used to trigger maintenance ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
