potiuk commented on PR #50057:
URL: https://github.com/apache/airflow/pull/50057#issuecomment-2844421276
If you look closely - our tests had shown that there are a couple of
consequences of "fixing the common.messaging" . So I think it would be good to
summarize it here. Also for @kaxil -> to agree and realise the impact it will
have on Airflow releases:
* this is definitely the "way to go" for common.messaging - > the coupling
introduced by the previous approach would be very difficult to justify and
manage, and baking it in the "Providers manger" interface brings in all the
niceties we have there and follows all other integrations
* we have to add lower-binding to the common.messaging>=1.0.1 in amazon (and
in kafka next) - as the implementation expects to be discovered by the
common.messaging discovery mechanism
* also common.messaging should have apache-airflow>=3.0.1 - because the
discovery mechanism for common messaging is going to be only avaiable in
Airflow 3.0.1 (assuming we will merge that one to 3.0.1).
That leaves a little bit of doubt - should we merge it to 3.0.1 ("bugfix")
or should it be 3.1.0 ("feature") of airflow to discover common.messaging
providers. I'd lean towards "bug-fix" and merging it with 3.0.1, in a way this
is fixing the way how common.messaging works and Airflow 3.0.0 has a lot more
things that all but guarantee literally no-one will use it. And we had a
precedent for that - Airflow 2.0.0 had a bug in provider's manager interface
that would prevent evolution of providers and we implemented a fix in 2.0.1 (or
even 2.0.2 - can't remember) that fixed it by adding essentially new thing to
airflow. This is not really a "user" facing feature, and intention is to "fix"
things - so for me this is reeally a "fix" and merging it to 3.0.1 is something
I would strongly recommend. That also will allow us to implement a lot more
common.messaging interfaces way quicker without waiting for 3.1.0 (like the
Kafka one)
A small consequence of that is that we have to add `"common.messaging" to
exclude it for comatibility matrix tests - for Airflow 3.0.0, but that's fine -
and we will remove it from that matrix once we bump the compatibility tests to
3.0.1.
So all seems on a clear path to be cherry-picked to 3.0.1 I think (and the
code changed in airflo is really a copy&paste /extension of the other provider
manager's features, so that is rather safe to cherry-pick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
