I think I'd just +1 Elad's comments. I don't know if we (the community) really need to be gatekeeping which providers get first class status like that. In the end, the users of any given provider become responsible for maintaining it, so I feel it sorts itself out without added bureaucracy. Perhaps some form of formalized decision tree on when to drop a provider package as "no longer maintained/supported", but I don't feel there should be a high barrier to entry on adding a new one provided the code doesn't break any existing packages and meets community quality standards.
________________________________ From: Elad Kalif <elad...@apache.org> Sent: Monday, April 4, 2022 7:24 AM To: dev@airflow.apache.org Subject: RE: [EXTERNAL] [DISCUSS] Approach for new providers of the community CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Interesting topic! I think the most important thing for us is that we are able to maintain the provider (in terms of not causing problems for Airflow core or other providers). Some of the maintained providers (Google for example) have open bugs for 2 years. So even if we have many provider mantiners it doesn't guarantee fixing problems. I am not worried about provider internal issues (operator not working properly, etc..) - it affects only the users of the provider itself and the users of the provider are always welcome to submit PRs with fixes. I don't feel comfortable blocking a new provider just because it has a small market / competitors' tools also don't support it etc... I guess my take is: 1. Accept any new provider that meets quality/requirements (just as we did so far) 2. Since providers are independent packages - In the rare case (I say rare as it never happened till now) where the provider causes problems with core/other providers and no one is willing to address it. if we can terminate the provider/mark it as not matinined in PyPi - it should be enough I think. On Mon, Apr 4, 2022 at 4:39 PM Jarek Potiuk <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: Hey all, We seem to have an influx of new providers coming our way: * Delta Sharing: https://lists.apache.org/thread/kny1f23noqf1ssh7l9ys607m5wk3ff8c * Flyte: https://lists.apache.org/thread/b55g3gydgmqmhow6f7xzzbm5t0gmhs2x * Versatile Data Kit: https://lists.apache.org/thread/t1k3d0518v4kxz1pqsprdc78h0wxobg0 I think it might be a good idea to bring the discussion in one place (here) and decide on what our approach is for accepting new providers (the original discussion from Andon was focused mostly about VDK's case, but maybe we could work out a general approach and "guidelines" - what approach is best so that we do not have to discuss it separately for each proposal, but we have some more (or less) clear rules on when we think it's good to accept providers as community. Generally speaking we have two approaches: * providers managed by the Apache Airflow community * providers managed by 3rd-parties I think my email here, nicely summarizes what is in https://lists.apache.org/thread/6oomg5rlphxvc7xl0nccm3zdg18qv83n I tried to look for earlier devlist discussions about the subject (maybe someone can find it :), I think we have never formalized nor written down but I do recall some (slack??) discussions about it from the past. While we have no control/influence (and we do not want to have) for 3rd-party providers, we definitely have both for the community-managed ones - and there should be some rules defined to decide when we are "ok" to accept a provider. Not always having "more" providers in the "community" area is better. More often than not, code is a liability more often than an asset. >From those discussions I had I recall points such us: * likelihood of the provider being used by many users * possibility to test/support the providers by maintainers or dedicated "stakeholders" * quality of the code and following our expectations (docs/how to guides, unit/system test) * competing (?) with Airflow - there could be some providers of "competing" products maybe (I am not sure if this is a concern of ours) which we simply might decide to not maintain in the community I am happy to write it down and propose such rules revolving around those - but I would like to hear what people think first. What are your thoughts here? J
