zach-overflow commented on code in PR #59239: URL: https://github.com/apache/airflow/pull/59239#discussion_r2645729346
########## airflow-core/docs/authoring-and-scheduling/deferring.rst: ########## @@ -469,6 +469,55 @@ According to `benchmarks <https://github.com/apache/airflow/pull/58803#pullreque You can determine a suitable value for your deployment by creating a large number of triggers (for example, by triggering a Dag with many deferrable tasks) and observing both how the load is distributed across Triggerers in your environment and how long it takes for all Triggerers to pick up the triggers. +Controlling Triggerer Host Assignment Per Trigger +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. versionadded:: 3.2.0 + +Under some circumstances, it may be desirable to assign a Trigger to a specific subset of ``triggerer`` hosts. Some examples of when this might be desirable are: + +* In a multi-tenant Airflow system where you run a distinct set of ``triggerers`` per team. +* Running distinct sets of ``triggerers`` hosts, where each set of hosts are configured for different trigger operations (e.g. each set of triggerers may have different cloud permissions). + +To achieve trigger assignment, you may use the optional "trigger queues" feature. + +To use trigger queues, do the following: + +1. For a given group of ``triggerer`` hosts, add ``--consume-trigger-queues=<comma-separated string of queue names to consume from>`` to the Triggerers' startup CLI command. This option ensures the triggerer will only run ``trigger`` instances with a ``trigger_queue`` value in the provided list. +2. To ensure a given ``trigger`` instance is assigned to that set of ``triggerers``, assign your Trigger instance's ``trigger_queue`` value in the ``kwargs`` in your trigger to any value in the ``--consume-trigger-queues`` list. + +For example, let's say you are running two triggerer (Triggerers "X", and "Y") hosts with the following commands, then any ``trigger`` instance with +``trigger_queue`` set to either ``"alice"``, or ``"bob"``, would be exclusively run by triggerer "X"; whereas, any ``trigger`` instance with +``trigger_queue`` set to ``"test_q"`` would be exclusively run by triggerer "Y". + +.. code-block:: bash + + # triggerer "X" startup command + airflow triggerer --consume-trigger-queues=alice,bob Review Comment: I'm happy to rename it, my thinking was just to clarify to users that trigger queues are not derived from celery queues or edge queues, but for consistency that seems reasonable to rename to `--queues`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
