Big yes to "WDYT about documenting those policies in this way?". That would finally set the rules and process to add a new provider. It will make it clearer for anyone who's thinking of creating a new provider and potentially avoid question we already had in the past in this email list (even though every question is welcome).
On 2023-03-30, 7:42 AM, "Jarek Potiuk" <[email protected] <mailto:[email protected]>> wrote: Yep. I deliberately do not want to discuss (yet) a policy about "removing" providers. As discussed in this comment on https://github.com/apache/airflow/pull/30359#discussion_r1152808160 <https://github.com/apache/airflow/pull/30359#discussion_r1152808160> - attic is the way to go, and when we get there to remove some of the providers, we should formalize it :) Note "suspended" is not "removed" - it is quite far from one to the other I think. J. On Thu, Mar 30, 2023 at 2:23 PM Kaxil Naik <[email protected] <mailto:[email protected]>> wrote: > > We need some version of attic <https://attic.apache.org/> > <https://attic.apache.org/>> when retiring > providers that aren't maintained. > > On Thu, 30 Mar 2023 at 12:52, Kaxil Naik <[email protected] > <mailto:[email protected]>> wrote: > > > I think this is a good starting point, we can iterate over it, over time. > > > > * WDYT about documenting those policies in this way? > > > > Yes, absolutely > > > > * should we expect Integration tests from kafka provider? > > > > I am leaning towards YES. It would have been a strong YES if we had > > created separate provider repo but my only fear here is the explosion of > > provider versions (Kafka, Trino versions) that we would have to run on > > main, adding to our already long CI jobs. Selective checks should help it > > keep lightweight on the PRs though. > > > > Regards, > > Kaxil > > > > On Thu, 30 Mar 2023 at 09:31, Jarek Potiuk <[email protected] > > <mailto:[email protected]>> wrote: > > > >> Hello everyone, > >> > >> While discussing the "suspension" I realised that we don't have a > >> formally described process of what is needed to approve new providers. > >> TL;DR; I would like to describe it more formally in our policies > >> > >> We have discussed it with the mixed governance model for "service" > >> providers and in numerous emails, but we do not have a clear > >> description about "how to get a new provider in". > >> > >> I believe the general consensus is that: > >> > >> * for new providers that have a strong "backing" - in the form of a > >> 3rd-party (service usually) that is mostly interested in managing and > >> supporting such a provider and has the capabilities to take the > >> maintenance of the provider, the strong preference is that provider is > >> released outside of the community - by the service provider. and then > >> they are free to mention their provider in various registries and in > >> our https://airflow.apache.org/ecosystem/ > >> <https://airflow.apache.org/ecosystem/> page. I think we should > >> strongly state that this is the default mode and very strong > >> preference. > >> > >> * if the service provider has really good reasons and can convince us > >> to take on the maintenance (think the recent "Huawei Cloud" discussion > >> https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc > >> <https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc> ), > >> the minimum requirement (on top of the regular code review rules/unit > >> testing/documentation etc.) is that the provider adds system tests > >> covering the service and provide the dashboard similar to the AWS one > >> > >> https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards > >> > >> <https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards> > >> . > >> This will allow the release manager to make a decision on release > >> viability, additionally (after the lazy consensus is reached) we > >> reserve the right of suspending such service provider (assuming lazy > >> consensus/vote of the community) if such provider is holding us back. > >> > >> * for providers that are referring to non-services (mostly open source > >> integration) providers, I think we should agree on being much more > >> open here. A good example is Apache Kafka provider > >> https://github.com/apache/airflow/pull/30175/files > >> <https://github.com/apache/airflow/pull/30175/files> that Dylan works > >> on. I am generally very receptive to having Kafka providers in > >> (especially if we agree on suspension rules - we will have a way to > >> manage problematic providers this way). One question I am not so sure > >> (but I am inclined to say "yes") - do we require integration tests > >> for such providers - we already have mongo, pinot. trino, cassandra, > >> celery and kerberos and they are now nicely separated out > >> https://github.com/apache/airflow/actions/runs/4561458457/jobs/8047548089 > >> <https://github.com/apache/airflow/actions/runs/4561458457/jobs/8047548089> > >> . > >> They are a bit more involved, but we have good mechanisms that make it > >> relatively easy to plug in new integration tests. > >> > >> > >> In either case - I think any time a new provider is added (regardless > >> if it is a service one or not), LAZY CONSENSUS/VOTE should be cast on > >> the devlist. It is not enough to just approve a PR IMHO. > >> > >> * WDYT about documenting those policies in this way? > >> * should we expect Integration tests from kafka provider? > >> > >> I think if we explicitly spell it out it would make it easier for > >> anyone (like Huawei Cloud case) to understand the options they have > >> and how much investment they need to make upfront, rather than finding > >> out after making some investment, that they are expected to do much > >> more. > >> > >> J. > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> <mailto:[email protected]> > >> For additional commands, e-mail: [email protected] > >> <mailto:[email protected]> > >> > >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] <mailto:[email protected]> For additional commands, e-mail: [email protected] <mailto:[email protected]> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
