potiuk commented on PR #49938:
URL: https://github.com/apache/airflow/pull/49938#issuecomment-2844687589
Ah we already have the kwargs..... ... bad me .. I see the point now.
First of all: - no, absolutely we should not have list of avaliable
providers in `common.messaging` . That completely defeats the purpose of
common.messaging being a generic interface if we want to make list of providers
in the common.messaging.
What would happen if we want to add a new provider to support
common.messaging? Should we release a new common.messaging provider version?
That makes no sense and is completely against the "genericness" of it - also it
prevents others from developing their `common.messaging` capable provider
independently.
Other than optional extras that indicate the list of providers that we have
in community that support common.messaging - there should be ABSOLUTELY NO
DIFFERENCE between our providers, and 3rd-party providers implementing
common.messaging. So list of "possible" providers in the code of
common.messaging is generally out-of-the-question.
I think the way it works now is not really too confusing and pretty good
The `queue` should be enough and should be interpreted by the particular
MessageQueue implementation - i.e. if the queue matches SQS, it's up to
SQSMessageQueue to interpret the queue + kwargs. Ideally it will derive some
basic parameters that can be passed to the Trigger without passing extra args.
But yes - there might be other MessageQueue implementations that expect more.
And it should be up to the implemenation how to interpret - both queue and
kwargs.
For example I can imagine a PubSubMessageQueue implementation that takes
such a queue (not sure what is the right form - it's more of a logical
representation
```
queue="pubsub://project:subscription", kwargs = {"gcp_conn_id" =
"connection" }
```
In this case the queue could be parsed by the PubSubMessageQueue and
converted into appropriate trigger parameters, while allowing additional
trigger parameters to be passed by kwargs if they cannot be easily translated
into `queue` URI or where we would like to use `some` standardized form.
We just need to be very clear - for each implementation - what is the format
of the `queue` URI. I don't think this will be universally standard - because
basically there is no universal standard for globally recognized URI format for
message queues. Or maybe there are 5 competing standards or smth.. I think -
similarly as in case of connections (where we have AIRFLOW connection URI form
- for good or better) this should be done this way.
Two learning from connections though:
* we have to be VERY clear that queue is not universally standard and each
message queue should precisely describe what kind of URI and kwargs it expects
* we might consider supporting "json" form of the `queue` on top of the URI.
URI has certain limitations (especially around encoding) and eventually we
settled on both URI and JSON formats supported for connection for that reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
