vincbeck commented on code in PR #77046:
URL: https://github.com/apache/airflow/pull/77046#discussion_r4591423768
##########
providers/common/messaging/src/airflow/providers/common/messaging/triggers/msg_queue.py:
##########
@@ -68,20 +70,55 @@ class MessageQueueTrigger(BaseEventTrigger):
def __init__(
self,
*,
- queue: str | None = None,
scheme: str | None = None,
+ queue: str | None = None,
trigger_queue: str | None = None,
+ queue_uri: str | None = None,
**kwargs: Any,
) -> None:
- # Stored under our own name rather than through
`BaseEventTrigger.__init__(queue=...)` so this
- # works regardless of the installed airflow-core version, see `queue`
property below.
- self._trigger_queue = trigger_queue
+ if trigger_queue is not None:
+ warnings.warn(
+ "The `trigger_queue` parameter is deprecated and will be
removed in a future version. "
+ "Use `queue` instead to assign this trigger to a trigger
queue.",
+ AirflowProviderDeprecationWarning,
+ stacklevel=2,
+ )
- if queue is None and scheme is None:
- raise ValueError("Either `queue` or `scheme` parameter must be
provided.")
+ if queue_uri is not None:
+ warnings.warn(
+ "The `queue_uri` parameter is deprecated and will be removed
in future versions. "
+ "Use the `scheme` parameter instead and pass configuration as
keyword arguments to `MessageQueueTrigger`.",
+ AirflowProviderDeprecationWarning,
+ stacklevel=2,
+ )
- # For backward compatibility, queue takes precedence
- if queue is not None:
+ # Determine whether `queue` represents the trigger queue or the legacy
broker queue URI.
+ # When `scheme` or `queue_uri` is explicitly provided, `queue` is the
trigger queue.
+ # When neither is provided, `queue` is treated as the legacy broker
queue URI for backward compatibility.
+ if scheme is not None and queue_uri is None and queue is not None and
"://" in queue:
Review Comment:
Checking for `://` only works for the in-tree providers. With a third-party
provider whose `queue_matches` accepts an identifier without `://`,
`MessageQueueTrigger(scheme=..., queue=<id>)` used to match on the URI and now
silently puts the trigger in a trigger queue named after the identifier. Can
you use `any(provider.queue_matches(queue) for provider in
MESSAGE_QUEUE_PROVIDERS)` instead? If you normalize `queue` into `queue_uri`
once at the top like this, `queue` always means the trigger queue afterwards
and the `elif queue is not None` branch can go away.
##########
providers/common/messaging/src/airflow/providers/common/messaging/triggers/msg_queue.py:
##########
@@ -68,20 +70,55 @@ class MessageQueueTrigger(BaseEventTrigger):
def __init__(
self,
*,
- queue: str | None = None,
scheme: str | None = None,
+ queue: str | None = None,
trigger_queue: str | None = None,
+ queue_uri: str | None = None,
Review Comment:
Why add `queue_uri` if it is deprecated from day one? Old callers can keep
`queue=<uri>` and new ones should use `scheme`. It also means there is no way
to use URI matching with a trigger queue without getting a warning.
##########
providers/common/messaging/src/airflow/providers/common/messaging/triggers/msg_queue.py:
##########
@@ -50,12 +50,14 @@ class MessageQueueTrigger(BaseEventTrigger):
This makes it easy to switch providers without modifying the trigger.
:param scheme: The queue scheme (e.g., 'kafka', 'redis+pubsub', 'sqs').
Used for provider matching.
- :param queue: **Deprecated** The queue identifier (URI format). If
provided, this takes precedence over scheme parameter.
- This parameter is deprecated and will be removed in future versions.
Use the 'scheme' parameter instead.
- :param trigger_queue: Assign this trigger to a specific trigger queue (see
+ :param queue: Assign this trigger to a specific trigger queue (see
Review Comment:
Should we mention in `docs/triggers.rst` that `queue` assigns the trigger to
a trigger queue? The docs do not talk about it today.
##########
providers/common/messaging/tests/unit/common/messaging/triggers/test_msg_queue.py:
##########
@@ -238,18 +246,43 @@ def test_scheme_param_does_not_set_trigger_queue(self):
trigger = MessageQueueTrigger(scheme=PROVIDER_2_SCHEME)
assert trigger.queue is None
- @pytest.mark.usefixtures("collect_queue_param_deprecation_warning")
+ def test_queue_param_sets_trigger_queue_alongside_scheme(self):
+ """Test that the unified `queue` parameter sets the trigger queue when
used with scheme."""
+ trigger = MessageQueueTrigger(scheme=PROVIDER_2_SCHEME,
queue="my-trigger-queue")
+ assert trigger.scheme == PROVIDER_2_SCHEME
+ assert trigger.queue_uri is None
+ assert trigger.queue == "my-trigger-queue"
+
+
@pytest.mark.filterwarnings("ignore::airflow.exceptions.AirflowProviderDeprecationWarning")
+ def test_queue_param_sets_trigger_queue_alongside_queue_uri(self):
+ """Test that `queue` sets the trigger queue when used alongside
deprecated `queue_uri`."""
+ trigger = MessageQueueTrigger(queue_uri=PROVIDER_1_QUEUE,
queue="my-trigger-queue")
+ assert trigger.queue_uri == PROVIDER_1_QUEUE
+ assert trigger.queue == "my-trigger-queue"
+
+
@pytest.mark.filterwarnings("ignore::airflow.exceptions.AirflowProviderDeprecationWarning")
def
test_trigger_queue_param_sets_trigger_queue_alongside_deprecated_queue(self):
"""The deprecated `queue` (broker URI) parameter claims the `queue`
keyword, so trigger
Review Comment:
nit: this docstring is no longer true, `trigger_queue` is deprecated now
##########
providers/common/messaging/src/airflow/providers/common/messaging/triggers/msg_queue.py:
##########
@@ -68,20 +70,55 @@ class MessageQueueTrigger(BaseEventTrigger):
def __init__(
self,
*,
- queue: str | None = None,
scheme: str | None = None,
+ queue: str | None = None,
trigger_queue: str | None = None,
+ queue_uri: str | None = None,
**kwargs: Any,
) -> None:
- # Stored under our own name rather than through
`BaseEventTrigger.__init__(queue=...)` so this
- # works regardless of the installed airflow-core version, see `queue`
property below.
- self._trigger_queue = trigger_queue
+ if trigger_queue is not None:
+ warnings.warn(
+ "The `trigger_queue` parameter is deprecated and will be
removed in a future version. "
+ "Use `queue` instead to assign this trigger to a trigger
queue.",
+ AirflowProviderDeprecationWarning,
+ stacklevel=2,
+ )
- if queue is None and scheme is None:
- raise ValueError("Either `queue` or `scheme` parameter must be
provided.")
+ if queue_uri is not None:
+ warnings.warn(
+ "The `queue_uri` parameter is deprecated and will be removed
in future versions. "
+ "Use the `scheme` parameter instead and pass configuration as
keyword arguments to `MessageQueueTrigger`.",
+ AirflowProviderDeprecationWarning,
+ stacklevel=2,
+ )
- # For backward compatibility, queue takes precedence
- if queue is not None:
+ # Determine whether `queue` represents the trigger queue or the legacy
broker queue URI.
+ # When `scheme` or `queue_uri` is explicitly provided, `queue` is the
trigger queue.
+ # When neither is provided, `queue` is treated as the legacy broker
queue URI for backward compatibility.
+ if scheme is not None and queue_uri is None and queue is not None and
"://" in queue:
+ # Existing callers passing the broker URI as `queue` next to
`scheme` used to get URI
+ # matching; keep that instead of routing to a trigger queue named
after the URI.
+ warnings.warn(
+ "Passing the broker queue URI as `queue` is deprecated, use
`scheme` instead.",
+ AirflowProviderDeprecationWarning,
+ stacklevel=2,
+ )
+ queue_uri, queue = queue, None
+ if scheme is not None or queue_uri is not None:
+ if queue is not None and trigger_queue is not None:
+ raise ValueError(
+ "Cannot provide both `queue` and deprecated
`trigger_queue`. For more details on "
+ "trigger queues, see: "
+
"https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#controlling-triggerer-host-assignment-per-trigger"
+ )
+ self._trigger_queue = queue if queue is not None else trigger_queue
+ if queue_uri is not None:
+ self.queue_uri = queue_uri
+ self.scheme = None
+ else:
+ self.queue_uri = None
+ self.scheme = scheme
+ elif queue is not None:
warnings.warn(
"The `queue` parameter is deprecated and will be removed in
future versions. "
Review Comment:
This message is now wrong: `queue` is no longer deprecated, only passing a
broker URI through it is. Users reading it will avoid `queue` for trigger
routing, which is the opposite of what this PR wants.
```suggestion
"Passing the broker queue URI as `queue` is deprecated and
will be removed in future versions. "
```
##########
providers/common/messaging/src/airflow/providers/common/messaging/triggers/msg_queue.py:
##########
@@ -68,20 +70,55 @@ class MessageQueueTrigger(BaseEventTrigger):
def __init__(
self,
*,
- queue: str | None = None,
scheme: str | None = None,
+ queue: str | None = None,
trigger_queue: str | None = None,
+ queue_uri: str | None = None,
**kwargs: Any,
) -> None:
- # Stored under our own name rather than through
`BaseEventTrigger.__init__(queue=...)` so this
- # works regardless of the installed airflow-core version, see `queue`
property below.
- self._trigger_queue = trigger_queue
+ if trigger_queue is not None:
+ warnings.warn(
+ "The `trigger_queue` parameter is deprecated and will be
removed in a future version. "
+ "Use `queue` instead to assign this trigger to a trigger
queue.",
Review Comment:
nit: in the legacy path (no `scheme`), `queue` already holds the broker URI,
so "Use `queue` instead" cannot be followed as is. Maybe say to use `scheme`
together with `queue`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]