jscheffl commented on code in PR #59239: URL: https://github.com/apache/airflow/pull/59239#discussion_r2645759790
########## airflow-core/docs/authoring-and-scheduling/deferring.rst: ########## @@ -469,6 +469,55 @@ According to `benchmarks <https://github.com/apache/airflow/pull/58803#pullreque You can determine a suitable value for your deployment by creating a large number of triggers (for example, by triggering a Dag with many deferrable tasks) and observing both how the load is distributed across Triggerers in your environment and how long it takes for all Triggerers to pick up the triggers. +Controlling Triggerer Host Assignment Per Trigger +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. versionadded:: 3.2.0 + +Under some circumstances, it may be desirable to assign a Trigger to a specific subset of ``triggerer`` hosts. Some examples of when this might be desirable are: + +* In a multi-tenant Airflow system where you run a distinct set of ``triggerers`` per team. +* Running distinct sets of ``triggerers`` hosts, where each set of hosts are configured for different trigger operations (e.g. each set of triggerers may have different cloud permissions). + +To achieve trigger assignment, you may use the optional "trigger queues" feature. + +To use trigger queues, do the following: + +1. For a given group of ``triggerer`` hosts, add ``--consume-trigger-queues=<comma-separated string of queue names to consume from>`` to the Triggerers' startup CLI command. This option ensures the triggerer will only run ``trigger`` instances with a ``trigger_queue`` value in the provided list. +2. To ensure a given ``trigger`` instance is assigned to that set of ``triggerers``, assign your Trigger instance's ``trigger_queue`` value in the ``kwargs`` in your trigger to any value in the ``--consume-trigger-queues`` list. + +For example, let's say you are running two triggerer (Triggerers "X", and "Y") hosts with the following commands, then any ``trigger`` instance with +``trigger_queue`` set to either ``"alice"``, or ``"bob"``, would be exclusively run by triggerer "X"; whereas, any ``trigger`` instance with +``trigger_queue`` set to ``"test_q"`` would be exclusively run by triggerer "Y". + +.. code-block:: bash + + # triggerer "X" startup command + airflow triggerer --consume-trigger-queues=alice,bob Review Comment: As far as I understood it would match 100% the queue used in Celery, correct? So if I use `queue_a` in celery (and not default) then it would be the same queue on triggerer later for all deferred tasks? Then it would in-deed make sense to have naming consistent. If it is not derived from the field we should re-consider the name in general because the we would confuse users with different queue names (which might be teams in these cases?) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
