This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 431a0608068 Add information about chicken-egg providers in
contributing docs (#44720)
431a0608068 is described below
commit 431a0608068fa68a9fe7a6e17e8f7f87f863dc0d
Author: Jarek Potiuk <[email protected]>
AuthorDate: Fri Dec 6 16:33:17 2024 +0100
Add information about chicken-egg providers in contributing docs (#44720)
* Add information about chicken-egg providers in contributing docs
Chicken-egg providers have not been fully described - the case
where two providers depend on each other have not been mentioned as
a use case in release management docs, also the providers have not
been mentioned in provider's contributing docs.
This PR addresses both problems.
* Update contributing-docs/11_provider_packages.rst
Co-authored-by: Amogh Desai <[email protected]>
* Update contributing-docs/11_provider_packages.rst
Co-authored-by: Amogh Desai <[email protected]>
---------
Co-authored-by: Amogh Desai <[email protected]>
---
contributing-docs/11_provider_packages.rst | 19 ++++++++++++++++---
dev/README_RELEASE_PROVIDER_PACKAGES.md | 9 +++++++++
2 files changed, 25 insertions(+), 3 deletions(-)
diff --git a/contributing-docs/11_provider_packages.rst
b/contributing-docs/11_provider_packages.rst
index e96d5e40d41..422adb09c2c 100644
--- a/contributing-docs/11_provider_packages.rst
+++ b/contributing-docs/11_provider_packages.rst
@@ -28,8 +28,8 @@ Airflow 2.0 is split into core and providers. They are
delivered as separate pac
Where providers are kept in our repository
------------------------------------------
-Airflow Providers are stored in the same source tree as Airflow Core (under
``airflow.providers``) package. This
-means that Airflow's repository is a monorepo, that keeps multiple packages in
a single repository. This has a number
+Airflow Providers are stored in a separate tree other than the Airflow Core
(under ``providers`` directory).
+Airflow's repository is a monorepo, that keeps multiple packages in a single
repository. This has a number
of advantages, because code and CI infrastructure and tests can be shared.
Also contributions are happening to a
single repository - so no matter if you contribute to Airflow or Providers,
you are contributing to the same
repository and project.
@@ -49,7 +49,7 @@ dependencies including runtime dependencies. See `local
virtualenv <../07_local_
for more information.
Therefore, until we can introduce multiple ``pyproject.toml`` for providers
information/meta-data about the providers
-is kept in ``provider.yaml`` file in the right sub-directory of
``airflow\providers``. This file contains:
+is kept in ``provider.yaml`` file in the right sub-directory of ``providers``.
This file contains:
* package name (``apache-airflow-provider-*``)
* user-facing name of the provider package
@@ -89,6 +89,19 @@ and pre-commit will generate new entry in
``generated/provider_dependencies.json
``pyproject.toml`` so that the package extra dependencies are properly handled
when package
might be installed when breeze is restarted or by your IDE or by running ``pip
install -e ".[devel]"``.
+Chicken-egg providers
+---------------------
+
+Sometimes, when a provider depends on another provider, and you want to add a
new feature that spans across
+two providers, you might need to add a new feature to the "dependent"
provider, you need
+to add a new feature to the "dependency" provider as well. This is a
chicken-egg problem and by default
+some CI jobs (like generating PyPI constraints) will fail because they cannot
use the source version of
+the provider package. This is handled by adding the "dependent" provider to
the chicken-egg list of
+"providers" in ``dev/breeze/src/airflow_breeze/global_constants.py``. By doing
this, the provider is build
+locally from sources rather than downloaded from PyPI when generating
constraints.
+
+More information about the chicken-egg providers and how release is handled
can be found in
+the `Release Provider Packages documentation
<../dev/README_RELEASE_PROVIDER_PACKAGES.md#chicken-egg-providers>`_
Developing community managed provider packages
----------------------------------------------
diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md
b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index 9e54a4b27d5..0afd99c77b6 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -203,6 +203,15 @@ release should get it changed to `>= x.y.z`. This is a
rare case and should be h
We call such case chicken-egg providers as it's not clear who should be
released first - the provider or
the Airflow.
+Similar case is when provider depends on another provider (usually `common.*`)
that is not yet released
+because you it contains new feature and you want to release the providers
together.
+
+In such case the "common" provider should be added to the list of
"chicken-egg" providers in the
+`./dev/breeze/src/airflow_breeze/global_constants.py` file and version of the
provider should be bumped
+in the PR by author of the PR that adds both the functionality. This will make
sure that the
+provider is build in CI from sources, when CI jobs are run rather than latest
version downloaded from PyPI
+when constraints are generated.
+
# Prepare Regular Provider packages (RC)
## Move provider into remove state
