Hello Everyone, TL;DR; I wanted to raise a discussion and make a proposal about option to skip some niche providers of our from releasing if they are holding us back, regarding the dependencies
We are going through some troubles with dependencies of our providers - mostly around some outdated dependencies which are used for some - often niche - providers of ours. We mitigated the problem for a while but now when Google team is working on a major bump of all the dependencies of google provider: https://github.com/apache/airflow/pull/30067 we have some other non-google providers that holds us back from upgrading those. This is mainly about the protobuf dependency (all the new google dependencies will have protobuf > 4 and there are few dependencies that have protobuf < 4. There are few ways we can deal with this in the order of approach: 1) we can replace dependencies with other dependencies which have the problem removed 2) we can (and we do) raise the issue with the respective dependencies 3) If the feature, the provider depends on is somewhat optional - we can make it so 4) finally if those are not successful we can disable the provider from further releases (until the problem is fixed) The first 3 are not really something we need to decide on specifically (and we are going to individually work on fixing those if possible). But I have a question, if we are ok to apply 4) Good example to look at is yandex provider - I just opened an issue for it whether they are planning to lift the limit to protobuf: https://github.com/yandex-cloud/python-sdk/issues/71 I think if we do not get a response and update in a few days/week we might need to disable the provider from next releases and testing. This is just an example, we might have other cases similar, I just wanted to discuss our approach there. What I propose is that in this case: a) we deliberately mark the provider as not maintained any more (that will include documentation update markingi it so b) we remove it from contributing to our dependencies and generating our constraints c) we stop running any tests with it d) old relasess will of course remain and the users will be able to use it for as long as they will be able to keep it conflict-free (but our constraints will not help with it) This will help us to move forward with some dependencies, but not being held-back by them. WDYT? J. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
