kaxil commented on code in PR #60896:
URL: https://github.com/apache/airflow/pull/60896#discussion_r2714828085
##########
PROVIDERS.rst:
##########
@@ -88,49 +88,233 @@ Airflow main branch to being decommissioned and removed
from the main branch in
`Managing provider's lifecycle
<https://github.com/apache/airflow/blob/main/providers/MANAGING_PROVIDERS_LIFECYCLE.rst>`_
+Provider Governance Framework
+-----------------------------
+
+This section describes the governance framework for community providers,
including lifecycle stages,
+stewardship model, and quantitative health metrics.
+
+Airflow's success is built on its extensive ecosystem of community-supported
integrations—over 1,600
+hooks, operators, and sensors released as part of 90+ provider packages. These
integrations are critical
+for "ubiquitous orchestration." This governance framework establishes a
scalable method to grow the
+number of integrations while ensuring quality. All governance remains with the
Airflow PMC.
+
+Provider Stewardship Model
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Each provider or integration component must have designated **stewards**
responsible for ensuring
+the health criteria described below are met:
+
+* **Minimum stewards**: At least two unique individuals must serve as stewards
for each provider
+* **Role definition**: Stewards are subject matter experts for the integration
(service, language,
+ or skill). Stewardship is a responsibility, not an additional authority or
privilege
+* **Committer sponsorship**: Each steward must be sponsored by an existing
Airflow Committer who
+ ensures stewards fulfill their responsibilities. The sponsor is responsible
for guiding the stewards
+ through the process of maintaining the provider, following best practices
including regular dependency
+ updates, issue resolution, and ensuring the provider meets the health
metrics described below. The sponsor
+ is NOT responsible for the codebase itself, which remains the responsibility
of the stewards.
Review Comment:
The sponsorship model here says "Each steward must be sponsored" but later
in the "Prerequisites" section (line 278) it says "At least one existing
Airflow Committer willing to sponsor the stewards" (plural, implying one
sponsor for all). The [AIP-95
text](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-95+Provider+lifecycle+update+proposal)
states: "each Steward is sponsored by an existing Airflow committer." Consider
making this consistent throughout—either one sponsor per steward or clarify
that one committer can sponsor multiple stewards.
##########
PROVIDERS.rst:
##########
@@ -88,49 +88,233 @@ Airflow main branch to being decommissioned and removed
from the main branch in
`Managing provider's lifecycle
<https://github.com/apache/airflow/blob/main/providers/MANAGING_PROVIDERS_LIFECYCLE.rst>`_
+Provider Governance Framework
+-----------------------------
+
+This section describes the governance framework for community providers,
including lifecycle stages,
+stewardship model, and quantitative health metrics.
+
+Airflow's success is built on its extensive ecosystem of community-supported
integrations—over 1,600
+hooks, operators, and sensors released as part of 90+ provider packages. These
integrations are critical
+for "ubiquitous orchestration." This governance framework establishes a
scalable method to grow the
+number of integrations while ensuring quality. All governance remains with the
Airflow PMC.
+
+Provider Stewardship Model
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Each provider or integration component must have designated **stewards**
responsible for ensuring
+the health criteria described below are met:
+
+* **Minimum stewards**: At least two unique individuals must serve as stewards
for each provider
+* **Role definition**: Stewards are subject matter experts for the integration
(service, language,
+ or skill). Stewardship is a responsibility, not an additional authority or
privilege
+* **Committer sponsorship**: Each steward must be sponsored by an existing
Airflow Committer who
+ ensures stewards fulfill their responsibilities. The sponsor is responsible
for guiding the stewards
+ through the process of maintaining the provider, following best practices
including regular dependency
+ updates, issue resolution, and ensuring the provider meets the health
metrics described below. The sponsor
+ is NOT responsible for the codebase itself, which remains the responsibility
of the stewards.
+* **Accountability**: Ultimate accountability remains with the Airflow PMC and
Committers
+* **Transitions**: Neither sponsorship nor stewardship are roles in
perpetuity—they can be
+ transitioned to others based on mutual agreement and PMC approval
+
+.. note::
+
+ The quantitative criteria described below are aspirational. The PMC will
revisit these metrics
+ based on actual experience 6 months after they are established and
published.
Review Comment:
This accurately reflects the AIP-95 language. Consider specifying when the
6-month review period starts (e.g., "from when this governance framework is
officially adopted" or "from the date of first quarterly review") for clarity.
##########
PROVIDERS.rst:
##########
@@ -88,49 +88,233 @@ Airflow main branch to being decommissioned and removed
from the main branch in
`Managing provider's lifecycle
<https://github.com/apache/airflow/blob/main/providers/MANAGING_PROVIDERS_LIFECYCLE.rst>`_
+Provider Governance Framework
+-----------------------------
+
+This section describes the governance framework for community providers,
including lifecycle stages,
+stewardship model, and quantitative health metrics.
+
+Airflow's success is built on its extensive ecosystem of community-supported
integrations—over 1,600
+hooks, operators, and sensors released as part of 90+ provider packages. These
integrations are critical
+for "ubiquitous orchestration." This governance framework establishes a
scalable method to grow the
+number of integrations while ensuring quality. All governance remains with the
Airflow PMC.
+
+Provider Stewardship Model
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Each provider or integration component must have designated **stewards**
responsible for ensuring
+the health criteria described below are met:
+
+* **Minimum stewards**: At least two unique individuals must serve as stewards
for each provider
+* **Role definition**: Stewards are subject matter experts for the integration
(service, language,
+ or skill). Stewardship is a responsibility, not an additional authority or
privilege
+* **Committer sponsorship**: Each steward must be sponsored by an existing
Airflow Committer who
+ ensures stewards fulfill their responsibilities. The sponsor is responsible
for guiding the stewards
+ through the process of maintaining the provider, following best practices
including regular dependency
+ updates, issue resolution, and ensuring the provider meets the health
metrics described below. The sponsor
+ is NOT responsible for the codebase itself, which remains the responsibility
of the stewards.
+* **Accountability**: Ultimate accountability remains with the Airflow PMC and
Committers
+* **Transitions**: Neither sponsorship nor stewardship are roles in
perpetuity—they can be
+ transitioned to others based on mutual agreement and PMC approval
+
+.. note::
+
+ The quantitative criteria described below are aspirational. The PMC will
revisit these metrics
+ based on actual experience 6 months after they are established and
published.
+
+Provider Lifecycle Stages
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+All providers follow a three-stage lifecycle: **Incubation**, **Production**,
and **Attic/Deprecation**.
+Additionally, providers may be designated as **Mature** under specific
circumstances.
+
+**Incubation Stage**
+
+All new providers or integration modules (such as Notifiers, Message
components, etc.) should start
+in incubation unless specifically accelerated by the PMC. Incubation ensures
that code, licenses, and
+processes align with ASF guidelines and community principles such as
community, collaborative
+development, and openness.
+
+Requirements for incubation:
+
+* Working codebase to bootstrap credibility and attract contributions
+* Visibility in the provider health dashboard
+* Designated stewards with committer sponsorship
+
+Quantitative graduation criteria:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 40 60
+
+ * - Metric
+ - Threshold
+ * - PRs submitted
+ - Minimum of 10 PRs in the last 6 months
+ * - PR review time
+ - PRs reviewed within 14 days
+ * - Issues reported
+ - Minimum of 15 unique issues filed in the last 6 months
+ * - Contributors
+ - At least 3 unique individuals making contributions (code or
documentation) in the last 6 months
+ * - Issue resolution rate
+ - At least 50% of reported issues closed within 90 days
+ * - Security/release issues
+ - All release and security related issues closed within 60 days
+ * - Governance participation
+ - Demonstrated participation in project governance channels including
quarterly updates
+ * - Quality standards
+ - Meet quality criteria for code, documentation, and tests as listed in
the Contributing Guide
+
+**Production Stage**
+
+All modules in production are expected to be well managed with prompt
resolution of issues and
+support for a consistent release cadence (at least monthly, but typically
every 2 weeks when changes
+exist). Production providers must:
+
+* Stay consistent with the main branch
+* Pass tests for main + all supported Airflow versions
+* Follow Airflow support guidelines, staying current with main Airflow releases
+
+Exceptions can be granted based on a PMC/devlist vote (PMC members only having
binding votes) for
+valid and one-off criteria.
+
+Quantitative criteria to maintain production status:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 40 60
+
+ * - Metric
+ - Threshold
+ * - PRs submitted
+ - Minimum of 10 PRs in the last 6 months
+ * - PR review time
+ - PRs reviewed within 14 days
+ * - Issues reported
+ - Minimum of 20 unique issues filed in the last 6 months
+ * - Contributors
+ - At least 5 unique individuals making contributions (code or
documentation) in the last 6 months
+ * - Issue resolution rate
+ - At least 60% of reported issues closed within 90 days
+ * - Security/release issues
+ - All release and security related issues closed within 30 days
+ * - Feature releases
+ - At least 1 feature release every 6 months
+ * - User engagement
+ - Maintain support activity with response to questions within 2 weeks on
average
+ * - Governance participation
+ - Demonstrated participation in project governance channels including
quarterly updates
+
+**Attic / Deprecation Stage**
+
+Modules should be moved into the Attic when relevance wanes, typically
measured by declining
+activity. This commonly occurs when the integrated solution has faded in
popularity and is replaced
+by more modern alternatives.
+
+Movement to the Attic must be communicated on the dev list and voted on by the
PMC. Exceptions can
+be granted based on the vote.
+
+Quantitative criteria triggering attic consideration:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 40 60
+
+ * - Metric
+ - Threshold
+ * - PRs submitted
+ - Fewer than 5 PRs in the last 6 months
+ * - PR review time
+ - PRs not being reviewed in more than a month
+ * - Issues reported
+ - Fewer than 10 unique issues filed in the last 6 months
+ * - Contributors
+ - Fewer than 3 unique individuals making contributions (code or
documentation) in the last 6 months
+ * - Issue resolution rate
+ - Less than 30% of reported issues closed within 90 days
+ * - Security/release issues
+ - Release and security related issues not getting closed within 30 days
+ * - Feature releases
+ - No feature releases in the last 6 months
+
+Consequences of attic status:
+
+* Modules remain readable but do not receive active maintenance
+* After a period of at least 6 months in the attic, modules can be chosen for
removal with
+ appropriate communication (see `Removing community providers`_ below)
+
+**Mature Providers**
+
+Some providers may have very stable interfaces requiring minimal changes on a
regular basis (e.g.,
+Slack provider integration). At the discretion of the Airflow PMC, certain
providers can be tagged
+as **"mature providers"**, which will not automatically be deprecated and
moved into the attic due
+to lack of activity alone.
+
+Periodic Reviews
+^^^^^^^^^^^^^^^^
+
+The Airflow PMC is responsible for reviewing the health status of integrations
on a **quarterly
+basis** and making decisions such as:
+
+* Graduating providers from incubation to production
+* Moving providers from production to the attic
+* Granting exceptions for specific providers
+
+These discussions will be held in public and subsequently summarized and
shared on the Airflow devlist.
+
+
Accepting new community providers
---------------------------------
-Accepting new community providers should be a deliberate process that requires
``[DISCUSSION]``
-followed by ``[VOTE]`` thread at the Airflow `devlist
<https://airflow.apache.org/community/#mailing-list>`_.
-
-In case the provider is integration with an open-source software rather than
service we can relax the vote
-procedure a bit. Particularly if the open-source software is an Apache
Software Foundation,
-Linux Software Foundation or similar organisation with well established
governance processes that are not
-strictly vendor-controlled, and when the software is well established an
popular, it might be enough to
-have a good and complete PR of the provider, ideally with a great test
coverage, including integration tests,
-and documentation. Then it should be enough to request the provider acceptance
by a ``[LAZY CONSENSUS]`` mail
-on the devlist and assuming such lazy consensus is not objected by anyone in
the community, the provider
-might be merged.
-
-For service providers, the ``[DISCUSSION]`` thread is aimed to gather
information about the reasons why
-the one who proposes the new provider thinks it should be accepted by the
community. Maintaining the provider
-in the community is a burden. Contrary to many people's beliefs, code is often
liability rather than asset,
-and accepting the code to be managed by the community, especially when it
involves significant effort on
-maintenance is often undesired, especially that the community consists of
volunteers. There must be a really
-good reason why we would believe that the provider is better to be maintained
by the community if there
-are 3rd-party teams that can be paid to manage it on their own. We have to
believe that the current
-community interest is in managing the provider and that enough volunteers in
the community will be
-willing to maintain it in the future in order to accept the provider.
-
-The ``[VOTE]`` thread is aimed to gather votes from the community on whether
the provider should be accepted
-or not and it follows the usual Apache Software Foundation voting rules
concerning
+The Airflow community welcomes new provider contributions. All new providers
enter through the
+**Incubation** stage (unless specifically accelerated by the PMC) to ensure
alignment with ASF
+guidelines and community principles.
+
+**Prerequisites for proposing a new provider:**
+
+1. **Working codebase**: A functional implementation demonstrating the
integration
+2. **Stewardship commitment**: At least two individuals willing to serve as
stewards
+3. **Committer sponsorship**: At least one existing Airflow Committer willing
to sponsor the stewards
+4. **Quality standards**: Code, tests, and documentation meeting the
Contributing Guide standards
+
+**Approval process:**
+
+Accepting new community providers requires a ``[DISCUSSION]`` followed by
``[VOTE]`` thread at the
+Airflow `devlist <https://airflow.apache.org/community/#mailing-list>`_.
+
+For integrations with well-established open-source software (Apache Software
Foundation, Linux
+Foundation, or similar organizations with established governance), a ``[LAZY
CONSENSUS]`` process
+may be sufficient, provided the PR includes comprehensive test coverage and
documentation.
+
+The ``[DISCUSSION]`` thread should include:
+
+* Description of the integration and its value to the Airflow ecosystem
+* Identification of the proposed stewards and their sponsoring committer(s)
+* Commitment to meet incubation health metrics within 6 months
+* Plan for participating in quarterly governance updates
+
+The ``[VOTE]`` follows the usual Apache Software Foundation voting rules
concerning
`Votes on Code Modification
<https://www.apache.org/foundation/voting.html#votes-on-code-modification>`_
-The Ecosystem page and registries, and own resources of the 3rd-party teams
are the best places to increase
-visibility that such providers exist, so there is no "great" visibility
achieved by getting the provider in
-the community. Also it is often easier to advertise and promote usage of the
provider by the service providers
-themselves when they own, manage and release their provider, especially when
they can synchronize releases
-of their provider with new feature, the service might get added.
+**Alternative: 3rd-party managed providers**
+
+For service providers or systems integrators with dedicated teams to manage
their provider and who wish to not participate
+in the Airflow community, we encourage considering 3rd-party management of
providers. The
+`Ecosystem page
<https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers>`_
+provides visibility for 3rd-party providers, and this approach allows service
providers or systems integrators to:
-Examples:
+* Synchronize releases with their service updates
+* Maintain direct control over the integration
+* Support older Airflow versions if needed
-Huawei Cloud provider - `Discussion
<https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc>`_
-Cloudera provider - `Discussion
<https://lists.apache.org/thread/2z0lvgj466ksxxrbvofx41qvn03jrwwb>`_, `Vote
<https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm>`_
-PgVector / Weaviate/ OpenAI provider - `Discussion
<https://lists.apache.org/thread/0d669fmy4hn29h5c0wj0ottdskd77ktp>`_, `Lazy
Consensus vote
<https://lists.apache.org/thread/zrq6554lwobhngtwyzp7tpgnyfsxxybh>`_
-Pinecone / OpenAI / Cohere provider - `Discussion
<https://lists.apache.org/thread/0d669fmy4hn29h5c0wj0ottdskd77ktp>`_, `Vote
<https://lists.apache.org/thread/skh32jksvcf4yx4fhhsfz8lq6w5nhfjc>`_, `VOTE
Result <https://lists.apache.org/thread/oq7h2n88zfo3dzldy5w8xlp9kyngs7x8>`_
+There is no difference in technical capabilities between community and
3rd-party providers.
+
+**Historical examples:**
+
+* Huawei Cloud provider - `Discussion
<https://lists.apache.org/thread/f5tk9c734wlyv616vyy8r34ymth3dqbc>`_
+* Cloudera provider - `Discussion
<https://lists.apache.org/thread/2z0lvgj466ksxxrbvofx41qvn03jrwwb>`_, `Vote
<https://lists.apache.org/thread/8b1jvld3npgzz2z0o3gv14lvtornbdrm>`_
Review Comment:
The previous version included additional historical examples
(PgVector/Weaviate/OpenAI, Pinecone/OpenAI/Cohere) that demonstrated the LAZY
CONSENSUS process. Consider keeping those as they provide useful precedents for
future contributors.
##########
PROVIDERS.rst:
##########
@@ -88,49 +88,233 @@ Airflow main branch to being decommissioned and removed
from the main branch in
`Managing provider's lifecycle
<https://github.com/apache/airflow/blob/main/providers/MANAGING_PROVIDERS_LIFECYCLE.rst>`_
+Provider Governance Framework
+-----------------------------
+
+This section describes the governance framework for community providers,
including lifecycle stages,
+stewardship model, and quantitative health metrics.
+
+Airflow's success is built on its extensive ecosystem of community-supported
integrations—over 1,600
+hooks, operators, and sensors released as part of 90+ provider packages. These
integrations are critical
+for "ubiquitous orchestration." This governance framework establishes a
scalable method to grow the
+number of integrations while ensuring quality. All governance remains with the
Airflow PMC.
+
+Provider Stewardship Model
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Each provider or integration component must have designated **stewards**
responsible for ensuring
+the health criteria described below are met:
+
+* **Minimum stewards**: At least two unique individuals must serve as stewards
for each provider
+* **Role definition**: Stewards are subject matter experts for the integration
(service, language,
+ or skill). Stewardship is a responsibility, not an additional authority or
privilege
+* **Committer sponsorship**: Each steward must be sponsored by an existing
Airflow Committer who
+ ensures stewards fulfill their responsibilities. The sponsor is responsible
for guiding the stewards
+ through the process of maintaining the provider, following best practices
including regular dependency
+ updates, issue resolution, and ensuring the provider meets the health
metrics described below. The sponsor
+ is NOT responsible for the codebase itself, which remains the responsibility
of the stewards.
+* **Accountability**: Ultimate accountability remains with the Airflow PMC and
Committers
+* **Transitions**: Neither sponsorship nor stewardship are roles in
perpetuity—they can be
+ transitioned to others based on mutual agreement and PMC approval
+
+.. note::
+
+ The quantitative criteria described below are aspirational. The PMC will
revisit these metrics
+ based on actual experience 6 months after they are established and
published.
+
+Provider Lifecycle Stages
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+All providers follow a three-stage lifecycle: **Incubation**, **Production**,
and **Attic/Deprecation**.
+Additionally, providers may be designated as **Mature** under specific
circumstances.
+
+**Incubation Stage**
+
+All new providers or integration modules (such as Notifiers, Message
components, etc.) should start
+in incubation unless specifically accelerated by the PMC. Incubation ensures
that code, licenses, and
+processes align with ASF guidelines and community principles such as
community, collaborative
+development, and openness.
+
+Requirements for incubation:
+
+* Working codebase to bootstrap credibility and attract contributions
+* Visibility in the provider health dashboard
Review Comment:
AIP-95 mentions building a dashboard as part of the updated technical
procedures, but it does not exist yet. Consider adding a note like "(planned)"
or clarifying that this is a future deliverable to avoid confusion for
contributors reading this now.
##########
providers/MANAGING_PROVIDERS_LIFECYCLE.rst:
##########
@@ -18,6 +18,36 @@
**The outline for this document in GitHub is available at top-right corner
button (with 3-dots and 3 lines).**
+Provider Governance Overview
+============================
+
+Before diving into the technical details of creating and managing providers,
please familiarize
+yourself with the governance framework that applies to all community providers:
+
+* `Provider Governance Framework
<https://github.com/apache/airflow/blob/main/PROVIDERS.rst#provider-governance-framework>`_
+
+Key governance concepts:
+
+* **Lifecycle stages**: All providers move through Incubation → Production →
Attic/Deprecation
+* **Stewardship**: Each provider requires at least two stewards sponsored by
Airflow Committers
+* **Health metrics**: Quantitative criteria determine readiness for promotion
or deprecation
+* **Periodic reviews**: The PMC reviews provider health quarterly
+
+All new providers must start in the **Incubation** stage (unless specifically
accelerated by the PMC)
+and meet the graduation criteria before moving to Production status.
+
+Stewardship Requirements for New Providers
+------------------------------------------
+
+When proposing a new community provider, you must:
+
+1. Identify at least two individuals willing to serve as stewards
+2. Secure sponsorship from at least one existing Airflow Committer for each
steward
Review Comment:
This says "at least one existing Airflow Committer for each steward" but
PROVIDERS.rst line 278 says "At least one existing Airflow Committer willing to
sponsor the stewards" (one for all). The AIP-95 text says "each Steward is
sponsored by an existing Airflow committer." These should be consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
