I think this is a good starting point, we can iterate over it, over time. * WDYT about documenting those policies in this way?
Yes, absolutely * should we expect Integration tests from kafka provider? I am leaning towards YES. It would have been a strong YES if we had created separate provider repo but my only fear here is the explosion of provider versions (Kafka, Trino versions) that we would have to run on main, adding to our already long CI jobs. Selective checks should help it keep lightweight on the PRs though. Regards, Kaxil On Thu, 30 Mar 2023 at 09:31, Jarek Potiuk <[email protected]> wrote: > Hello everyone, > > While discussing the "suspension" I realised that we don't have a > formally described process of what is needed to approve new providers. > TL;DR; I would like to describe it more formally in our policies > > We have discussed it with the mixed governance model for "service" > providers and in numerous emails, but we do not have a clear > description about "how to get a new provider in". > > I believe the general consensus is that: > > * for new providers that have a strong "backing" - in the form of a > 3rd-party (service usually) that is mostly interested in managing and > supporting such a provider and has the capabilities to take the > maintenance of the provider, the strong preference is that provider is > released outside of the community - by the service provider. and then > they are free to mention their provider in various registries and in > our https://airflow.apache.org/ecosystem/ page. I think we should > strongly state that this is the default mode and very strong > preference. > > * if the service provider has really good reasons and can convince us > to take on the maintenance (think the recent "Huawei Cloud" discussion > https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc ), > the minimum requirement (on top of the regular code review rules/unit > testing/documentation etc.) is that the provider adds system tests > covering the service and provide the dashboard similar to the AWS one > > https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards > . > This will allow the release manager to make a decision on release > viability, additionally (after the lazy consensus is reached) we > reserve the right of suspending such service provider (assuming lazy > consensus/vote of the community) if such provider is holding us back. > > * for providers that are referring to non-services (mostly open source > integration) providers, I think we should agree on being much more > open here. A good example is Apache Kafka provider > https://github.com/apache/airflow/pull/30175/files that Dylan works > on. I am generally very receptive to having Kafka providers in > (especially if we agree on suspension rules - we will have a way to > manage problematic providers this way). One question I am not so sure > (but I am inclined to say "yes") - do we require integration tests > for such providers - we already have mongo, pinot. trino, cassandra, > celery and kerberos and they are now nicely separated out > https://github.com/apache/airflow/actions/runs/4561458457/jobs/8047548089. > They are a bit more involved, but we have good mechanisms that make it > relatively easy to plug in new integration tests. > > > In either case - I think any time a new provider is added (regardless > if it is a service one or not), LAZY CONSENSUS/VOTE should be cast on > the devlist. It is not enough to just approve a PR IMHO. > > * WDYT about documenting those policies in this way? > * should we expect Integration tests from kafka provider? > > I think if we explicitly spell it out it would make it easier for > anyone (like Huawei Cloud case) to understand the options they have > and how much investment they need to make upfront, rather than finding > out after making some investment, that they are expected to do much > more. > > J. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
