Hello everyone, While discussing the "suspension" I realised that we don't have a formally described process of what is needed to approve new providers. TL;DR; I would like to describe it more formally in our policies
We have discussed it with the mixed governance model for "service" providers and in numerous emails, but we do not have a clear description about "how to get a new provider in". I believe the general consensus is that: * for new providers that have a strong "backing" - in the form of a 3rd-party (service usually) that is mostly interested in managing and supporting such a provider and has the capabilities to take the maintenance of the provider, the strong preference is that provider is released outside of the community - by the service provider. and then they are free to mention their provider in various registries and in our https://airflow.apache.org/ecosystem/ page. I think we should strongly state that this is the default mode and very strong preference. * if the service provider has really good reasons and can convince us to take on the maintenance (think the recent "Huawei Cloud" discussion https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc ), the minimum requirement (on top of the regular code review rules/unit testing/documentation etc.) is that the provider adds system tests covering the service and provide the dashboard similar to the AWS one https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards. This will allow the release manager to make a decision on release viability, additionally (after the lazy consensus is reached) we reserve the right of suspending such service provider (assuming lazy consensus/vote of the community) if such provider is holding us back. * for providers that are referring to non-services (mostly open source integration) providers, I think we should agree on being much more open here. A good example is Apache Kafka provider https://github.com/apache/airflow/pull/30175/files that Dylan works on. I am generally very receptive to having Kafka providers in (especially if we agree on suspension rules - we will have a way to manage problematic providers this way). One question I am not so sure (but I am inclined to say "yes") - do we require integration tests for such providers - we already have mongo, pinot. trino, cassandra, celery and kerberos and they are now nicely separated out https://github.com/apache/airflow/actions/runs/4561458457/jobs/8047548089. They are a bit more involved, but we have good mechanisms that make it relatively easy to plug in new integration tests. In either case - I think any time a new provider is added (regardless if it is a service one or not), LAZY CONSENSUS/VOTE should be cast on the devlist. It is not enough to just approve a PR IMHO. * WDYT about documenting those policies in this way? * should we expect Integration tests from kafka provider? I think if we explicitly spell it out it would make it easier for anyone (like Huawei Cloud case) to understand the options they have and how much investment they need to make upfront, rather than finding out after making some investment, that they are expected to do much more. J. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
