Yeah, I can see that. I'd vote plus-one, but for the sake of discussion: my only real (non-blocking) concern is that there may be some (perceived?) bias in the process if there isn't some kind of set policy. It inherently favors the providers of the companies that have dedicated contributors, for one. And if we flip the coin on your example and Yandex does respond and submits an update that Google doesn't yet support (ok, that's maybe a bit of a stretch, but bear with me) do we really suspend the bigger/popular provider?
As long as we are alright handling the potential issues like adults, then I think it's a solid idea. I've just seen too many groups of grown adults devolve into children when they perceive favoritism or bias. - ferruzzi ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Wednesday, March 29, 2023 11:06 AM To: dev@airflow.apache.org Subject: RE: [EXTERNAL][DISCUSS] Exclude some providers that hold us back from releasing CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. I see a consensus building up :). So I will formally call for a lazy consensus in a moment with PR where I will spell out the details. @dennis @danniel -> It will be difficult to get very predetermined criteria I think. There might be various reasons why we might think we decide that "enough is enough" for old dependencies (note that this is only for providers that cling to the dependencies where other providers want to upgrade to newer versions and they are held back - it is never in the other direction). But I think we do not have to have such criteria defined. Such change will be reversible and in the best case if the root cause of the problem is fixed quickly (whether with the involvement of a 3rd-party or not), we can always bring such providers back. This at most will result in not releasing new changes to it in the next wave of providers. But such changes won't happen - because in case of a provider that blocks us from doing something, changes to it will be rejected anyway as tests will be disabled - we will automate some of that). In this context I propose 1 week for notification. Should be enough, and since it is reversible. It's really a "soft" removal - so actually a "suspension" that can be "resumed". We are not going to remove the code (for now at least but we might have some criteria for that in the future) - so bringing a change which fixes it (usually by a dependency release that unblocks the conflict) should be enough to "resume" such a provider. PR/Lazy consensus follows. J. On Wed, Mar 29, 2023 at 3:33 PM Josh Fell <josh.d.f...@astronomer.io.invalid> wrote: > > Agreed, this seems like a good move. > > On Wed, Mar 29, 2023 at 9:19 AM Beck, Vincent <vincb...@amazon.com.invalid> > wrote: > > > +1 > > > > On 2023-03-29, 2:26 AM, "Elad Kalif" <elad...@apache.org <mailto: > > elad...@apache.org>> wrote: > > > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and know > > the content is safe. > > > > > > > > > > > > > > I agree that we should exclude providers that block us. > > > > > > > > > > On Tue, Mar 28, 2023 at 9:58 PM Ferruzzi, Dennis > > <ferru...@amazon.com.inva <mailto:ferru...@amazon.com.inva>lid> wrote: > > > > > > > I don't have any issue with this in general, in fact I think it's not a > > > bad idea to trim out older unused/unmaintained providers. But what are > > the > > > criteria for marking a provider package as unmaintained? Is it simply > > > "once a package becomes a blocker AND nobody has stepped up to fix it in > > > [two weeks?]". Also, to clarify, "unmaintained" in this context isn't > > > going to prevent current users from using it, it's just a notation > > > indicating that nobody is actively updating/upgrading it, correct? > > > > > > > > > - ferruzzi > > > > > > > > > ________________________________ > > > From: Ash Berlin-Taylor <a...@apache.org <mailto:a...@apache.org>> > > > Sent: Tuesday, March 28, 2023 7:11 AM > > > To: dev@airflow.apache.org <mailto:dev@airflow.apache.org>; Daniel > > Standish > > > Subject: RE: [EXTERNAL][DISCUSS] Exclude some providers that hold us back > > > from releasing > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > Yeah agreed. > > > > > > Each provider should be treated like a separate project/release - it's > > > just that we batch up releases to save time of the release manager. > > > > > > -a > > > > > > On 28 March 2023 07:05:13 BST, Daniel Standish > > > <daniel.stand...@astronomer.io.inva <mailto: > > daniel.stand...@astronomer.io.inva>LID> wrote: > > > >It seems reasonable to me. > > > > > > > >On Mon, Mar 27, 2023 at 12:02 AM Jarek Potiuk <ja...@potiuk.com > > <mailto:ja...@potiuk.com>> wrote: > > > > > > > >> Hello Everyone, > > > >> > > > >> TL;DR; I wanted to raise a discussion and make a proposal about option > > > >> to skip some niche providers of our from releasing if they are holding > > > >> us back, regarding the dependencies > > > >> > > > >> We are going through some troubles with dependencies of our providers > > > >> - mostly around some outdated dependencies which are used for some - > > > >> often niche - providers of ours. We mitigated the problem for a while > > > >> but now when Google team is working on a major bump of all the > > > >> dependencies of google provider: > > > >> https://github.com/apache/airflow/pull/30067 < > > https://github.com/apache/airflow/pull/30067> we have some other > > > >> non-google providers that holds us back from upgrading those. This is > > > >> mainly about the protobuf dependency (all the new google dependencies > > > >> will have protobuf > 4 and there are few dependencies that have > > > >> protobuf < 4. > > > >> > > > >> There are few ways we can deal with this in the order of approach: > > > >> > > > >> 1) we can replace dependencies with other dependencies which have the > > > >> problem removed > > > >> 2) we can (and we do) raise the issue with the respective dependencies > > > >> 3) If the feature, the provider depends on is somewhat optional - we > > > >> can make it so > > > >> 4) finally if those are not successful we can disable the provider > > > >> from further releases (until the problem is fixed) > > > >> > > > >> The first 3 are not really something we need to decide on specifically > > > >> (and we are going to individually work on fixing those if possible). > > > >> > > > >> But I have a question, if we are ok to apply 4) > > > >> > > > >> Good example to look at is yandex provider - I just opened an issue > > > >> for it whether they are planning to lift the limit to protobuf: > > > >> https://github.com/yandex-cloud/python-sdk/issues/71 < > > https://github.com/yandex-cloud/python-sdk/issues/71> I think if we do > > > >> not get a response and update in a few days/week we might need to > > > >> disable the provider from next releases and testing. This is just an > > > >> example, we might have other cases similar, I just wanted to discuss > > > >> our approach there. > > > >> > > > >> What I propose is that in this case: > > > >> > > > >> a) we deliberately mark the provider as not maintained any more (that > > > >> will include documentation update markingi it so > > > >> > > > >> b) we remove it from contributing to our dependencies and generating > > > >> our constraints > > > >> > > > >> c) we stop running any tests with it > > > >> > > > >> d) old relasess will of course remain and the users will be able to > > > >> use it for as long as they will be able to keep it conflict-free (but > > > >> our constraints will not help with it) > > > >> > > > >> This will help us to move forward with some dependencies, but not > > > >> being held-back by them. > > > >> > > > >> WDYT? > > > >> > > > >> J. > > > >> > > > >> --------------------------------------------------------------------- > > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org <mailto: > > dev-unsubscr...@airflow.apache.org> > > > >> For additional commands, e-mail: dev-h...@airflow.apache.org <mailto: > > dev-h...@airflow.apache.org> > > > >> > > > >> > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
