This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v3-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-1-test by this push:
new 628d7520e6c [v3-1-test] Fix documentation for installing from sources
(#58366) (#58373)
628d7520e6c is described below
commit 628d7520e6c86a440dc016ccfc989f45eb5f728c
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Sun Nov 16 17:45:22 2025 +0100
[v3-1-test] Fix documentation for installing from sources (#58366) (#58373)
There were a couple of problems with genearted documentation for
installation from sources for "apache-airflow" and "apache-airflow-ctl":
* apache-airflow-ctl links in source installation were broken
* apache-airflow did not have link to sdist and whl distributions
of task-sdk (which is currently released together with apache-airflow
and apache-airflow-core)
* airflow-ctl was missing in development index generated locally
* INSTALL that is common for all distribution sources only contained
information on building apache-airflow, but did not have
instrunctions on building other distributions
(cherry picked from commit f4c7dad2fb54e7b0027a66fde13d76b6c6a3a519)
Co-authored-by: Jarek Potiuk <[email protected]>
---
INSTALL | 64 +++++++++++++++++-----
airflow-core/docs/conf.py | 7 +++
.../docs/installation/installing-from-sources.rst | 13 +++--
airflow-ctl/docs/conf.py | 12 +++-
.../docs/installation/installing-from-sources.rst | 22 ++------
dev/README_RELEASE_AIRFLOWCTL.md | 4 +-
.../docs_build/dev_index_template.html.jinja2 | 25 ++++++---
docs/spelling_wordlist.txt | 1 +
8 files changed, 98 insertions(+), 50 deletions(-)
diff --git a/INSTALL b/INSTALL
index 7d4cc066d98..0a907c6e6a6 100644
--- a/INSTALL
+++ b/INSTALL
@@ -47,6 +47,21 @@ you get all sources in one place. This is the most
convenient way to develop Air
Otherwise, you have to install Airflow and Providers separately from sources
in the same environment, which
is not as convenient.
+
+Content of the source archive
+-----------------------------
+
+The archive contains a complete snapshot of the whole "apache-airflow"
repository, including all
+distributions that can be built from the sources: `apache-airflow`
meta-distribution (that you can use to install
+all other distributions, `apache-airflow-core`, `apache-airflow-task-sdk`,
`apache-airflow-ctl`,
+`apache-airflow-go-sdk`, more than 90 `apache-airflow-providers-<provider>`
distributions, and
+all the other distributions that are part of the monorepo and are needed to
build other packages,
+their documentation and also allows to run tests for all those distributions.
+
+We are using `uv` and workspace tooling to build and manage the packages
together, see below for more details.
+Whatever distribution you choose to install, you need to localise the right
`pyproject.toml` file in the
+repository and this is the one that you should use to build the distribution
you need.`
+
Using ``uv`` to manage your Python, virtualenvs, and install airflow for
development (recommended)
==================================================================================================
@@ -62,6 +77,7 @@ Installing ``uv``
You can install uv following the instructions:
https://docs.astral.sh/uv/getting-started/installation/
+
Using ``uv`` to manage your project dependencies
------------------------------------------------
@@ -107,8 +123,8 @@ You can run any command in the virtual environment created
by `uv` by prefixing
This will automatically synchronize your dependencies to latest dependencies
needed.
-Compiling front-end assets
---------------------------
+Compiling front-end assets for Airflow Core
+-------------------------------------------
In order to see UI in Airflow, you need to compile front-end assets first.
@@ -142,12 +158,25 @@ it in the UI) that should contain the git commit hash of
the build and it will g
The result of this command is airflow sdist package built in the `dist` folder
of `airflow-core`
package as well. It requires ``prek`` to be installed in your system.
+There are also similar ``prek`` commands for other packages in the repository
- for example:
+
+```
+compile-edge-assets -- Compile Edge provider assets
+compile-fab-assets -- Compile FAB provider assets
+```
+
+However, the compiled, generated assets for those are checked in the
repository and you do not need to
+compile them manually before building the packages - you only need to do it
when you modify the original
+UI files for those packages.
+
Using pip and manually managing your virtualenv
===============================================
-While `uv` manages dependencies and venv automatically you might want to
manage both manually with
-pip and virtualenv. You need to have Python installed in your preferred way
for that to work. It is also
-way slower than with `uv` and you need to manage your environment manually.
+While `uv` manages dependencies and venv automatically and manage workspace
automatically, you might want
+to manage both manually with pip and virtualenv. You need to have Python
installed in your preferred
+way for that to work. It is also way slower than with `uv` and you need to
manage your environment manually
+and sometimes install several distributions together to make tests and
documentation work - emulating what
+workspace tooling does automatically.
Creating virtualenv
-------------------
@@ -193,16 +222,19 @@ It's important to keep your hatch up to date. You can do
this by running:
uv tool upgrade hatch
-Using Hatch to build your packages
-----------------------------------
+Using Hatch to build packages
+-----------------------------
You can use Hatch to build installable packages from the Airflow sources. Such
package will
include all metadata configured in `pyproject.toml` and will be installable
with ``pip`` and and any other
PEP-compliant packaging front-end. You can run those commands in:
-* root folder of the repository to build "meta" airflow package that install
other packages
-* `airflow-core` folder to build the core airflow package
-* any of the `providers` folders that has a pyproject.toml file to build the
provider package
+* root folder of the repository to build "meta" airflow distribution that
install other distribution
+* `airflow-core` folder to build the core airflow distribution
+* any of the `providers` folders that has a pyproject.toml file to build the
provider distribution
+* task-sdk to build the task-sdk distribution
+* airflow-ctl to build the airflow-ctl distribution
+* task-go-sdk to build the task-go-sdk distribution
The packages will have pre-installed dependencies for providers that are
available when Airflow is i
onstalled from PyPI. Both `wheel` and `sdist` packages are built by default.
@@ -214,11 +246,13 @@ You can also build only `wheel` or `sdist` packages:
hatch build -t wheel
hatch build -t sdist
-In the `airflow-core` folder, you can also build the package with the `custom`
target that will clean
-the build directory, update the `git_version` file, and build the assets:
+In the `airflow-core` folder, you should also build the package with the
`custom` target that will clean
+the build directory, update the `git_version` file, and build the assets (in
case you have not built
+them already manually with `prek`):
hatch build -t custom -t wheel -t sdist
+
Installing recommended version of dependencies
==============================================
@@ -231,15 +265,15 @@ that are used in main CI tests and by other contributors.
There are different constraint files for different Python versions. For
example, this command will install
all basic devel requirements and requirements of Google provider as last
successfully tested for Python 3.10:
- uv pip install -e ".[devel,google]"" \
+ pip install -e ".[devel,google]"" \
--constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.10.txt";
Using the 'constraints-no-providers' constraint files, you can upgrade Airflow
without paying attention to the provider's dependencies. This allows you to
keep installed provider dependencies and install the latest supported ones
using pure Airflow core.
- uv pip install -e ".[devel]" \
+ pip install -e ".[devel]" \
--constraint
"https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.10.txt";
-Note that you can also use `pip install` if you do not use `uv`.
+Note that you can also use `uv pip install` if you use `uv`.
Airflow extras
==============
diff --git a/airflow-core/docs/conf.py b/airflow-core/docs/conf.py
index bd63dd2771f..74c09b31c7a 100644
--- a/airflow-core/docs/conf.py
+++ b/airflow-core/docs/conf.py
@@ -244,13 +244,20 @@ airflow_version: Version = parse_version(
config_descriptions =
retrieve_configuration_description(include_providers=False)
configs, deprecated_options = get_configs_and_deprecations(airflow_version,
config_descriptions)
+# TODO: remove it when we start releasing task-sdk separately from airflow-core
+airflow_version_split = PACKAGE_VERSION.split(".")
+TASK_SDK_VERSION = f"1.{airflow_version_split[1]}.{airflow_version_split[2]}"
+
jinja_contexts = {
"config_ctx": {"configs": configs, "deprecated_options":
deprecated_options},
"quick_start_ctx": {"doc_root_url":
f"https://airflow.apache.org/docs/apache-airflow/{PACKAGE_VERSION}/"},
"official_download_page": {
"base_url": f"https://downloads.apache.org/airflow/{PACKAGE_VERSION}";,
+ "base_url_task_sdk":
f"https://downloads.apache.org/airflow/task-sdk/{TASK_SDK_VERSION}";,
"closer_lua_url":
f"https://www.apache.org/dyn/closer.lua/airflow/{PACKAGE_VERSION}";,
+ "closer_lua_url_task_sdk":
f"https://www.apache.org/dyn/closer.lua/airflow/task-sdk/{TASK_SDK_VERSION}";,
"airflow_version": PACKAGE_VERSION,
+ "task_sdk_version": TASK_SDK_VERSION,
},
}
diff --git a/airflow-core/docs/installation/installing-from-sources.rst
b/airflow-core/docs/installation/installing-from-sources.rst
index ccb01b82c93..71272e81b96 100644
--- a/airflow-core/docs/installation/installing-from-sources.rst
+++ b/airflow-core/docs/installation/installing-from-sources.rst
@@ -23,8 +23,7 @@ Released packages
.. jinja:: official_download_page
- This page describes downloading and verifying Airflow® version
- ``{{ airflow_version }}`` using officially released packages.
+ This page describes downloading and verifying Airflow® version
``|version|`` using officially released packages.
You can also install ``Apache Airflow`` - as most Python packages - via
:doc:`PyPI <installing-from-pypi>`.
You can choose different version of Airflow by selecting a different
version from the drop-down at
the top-left of the page.
@@ -46,10 +45,12 @@ The |version| downloads of Airflow® are available at:
.. jinja:: official_download_page
* `Sources package for airflow <{{ closer_lua_url }}/apache_airflow-{{
airflow_version }}-source.tar.gz>`__ (`asc <{{ base_url }}/apache_airflow-{{
airflow_version }}-source.tar.gz.asc>`__, `sha512 <{{ base_url
}}/apache_airflow-{{ airflow_version }}-source.tar.gz.sha512>`__)
- * `Sdist package for airflow meta package <{{ closer_lua_url
}}/apache_airflow-{{ airflow_version }}.tar.gz>`__ (`asc <{{ base_url
}}/apache_airflow-{{ airflow_version }}.tar.gz.asc>`__, `sha512 <{{ base_url
}}/apache_airflow-{{ airflow_version }}.tar.gz.sha512>`__)
- * `Whl package for airflow meta package <{{ closer_lua_url
}}/apache_airflow-{{ airflow_version }}-py3-none-any.whl>`__ (`asc <{{ base_url
}}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.asc>`__, `sha512 <{{
base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.sha512>`__)
- * `Sdist package for airflow core package <{{ closer_lua_url
}}/apache_airflow_core-{{ airflow_version }}.tar.gz>`__ (`asc <{{ base_url
}}/apache_airflow_core-{{ airflow_version }}.tar.gz.asc>`__, `sha512 <{{
base_url }}/apache_airflow_core-{{ airflow_version }}.tar.gz.sha512>`__)
- * `Whl package for airflow core package <{{ closer_lua_url
}}/apache_airflow_core-{{ airflow_version }}-py3-none-any.whl>`__ (`asc <{{
base_url }}/apache_airflow_core-{{ airflow_version }}-py3-none-any.whl.asc>`__,
`sha512 <{{ base_url }}/apache_airflow_core-{{ airflow_version
}}-py3-none-any.whl.sha512>`__)
+ * `Sdist package for airflow meta distribution <{{ closer_lua_url
}}/apache_airflow-{{ airflow_version }}.tar.gz>`__ (`asc <{{ base_url
}}/apache_airflow-{{ airflow_version }}.tar.gz.asc>`__, `sha512 <{{ base_url
}}/apache_airflow-{{ airflow_version }}.tar.gz.sha512>`__)
+ * `Whl package for airflow meta distributio <{{ closer_lua_url
}}/apache_airflow-{{ airflow_version }}-py3-none-any.whl>`__ (`asc <{{ base_url
}}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.asc>`__, `sha512 <{{
base_url }}/apache_airflow-{{ airflow_version }}-py3-none-any.whl.sha512>`__)
+ * `Sdist package for airflow core distribution <{{ closer_lua_url
}}/apache_airflow_core-{{ airflow_version }}.tar.gz>`__ (`asc <{{ base_url
}}/apache_airflow_core-{{ airflow_version }}.tar.gz.asc>`__, `sha512 <{{
base_url }}/apache_airflow_core-{{ airflow_version }}.tar.gz.sha512>`__)
+ * `Whl package for airflow core distribution <{{ closer_lua_url
}}/apache_airflow_core-{{ airflow_version }}-py3-none-any.whl>`__ (`asc <{{
base_url }}/apache_airflow_core-{{ airflow_version }}-py3-none-any.whl.asc>`__,
`sha512 <{{ base_url }}/apache_airflow_core-{{ airflow_version
}}-py3-none-any.whl.sha512>`__)
+ * `Sdist package for airflow task-sdk distribution <{{
closer_lua_url_task_sdk }}/apache_airflow_task_sdk-{{ task_sdk_version
}}.tar.gz>`__ (`asc <{{ base_url_task_sdk }}/apache_airflow_task_sdk-{{
task_sdk_version }}.tar.gz.asc>`__, `sha512 <{{ base_url_task_sdk
}}/apache_airflow_task_sdk-{{ task_sdk_version }}.tar.gz.sha512>`__)
+ * `Whl package for airflow task-sdk distribution <{{
closer_lua_url_task_sdk }}/apache_airflow_task_sdk-{{ task_sdk_version
}}-py3-none-any.whl>`__ (`asc <{{ base_url_task_sdk
}}/apache_airflow_task_sdk-{{ task_sdk_version }}-py3-none-any.whl.asc>`__,
`sha512 <{{ base_url_task_sdk }}/apache_airflow_task_sdk-{{ task_sdk_version
}}-py3-none-any.whl.sha512>`__)
If you want to install from the source code, you can download from the sources
link above, it will contain
a ``INSTALL`` file containing details on how you can build and install Airflow.
diff --git a/airflow-ctl/docs/conf.py b/airflow-ctl/docs/conf.py
index 08f867ee43f..5a19b2adea3 100644
--- a/airflow-ctl/docs/conf.py
+++ b/airflow-ctl/docs/conf.py
@@ -207,12 +207,18 @@ jinja_contexts = {
"config_ctx": {"configs": configs, "deprecated_options":
deprecated_options},
"quick_start_ctx": {"doc_root_url":
f"https://airflow.apache.org/docs/apache-airflow/{PACKAGE_VERSION}/"},
"official_download_page": {
- "base_url": f"https://downloads.apache.org/airflow/{PACKAGE_VERSION}";,
- "closer_lua_url":
f"https://www.apache.org/dyn/closer.lua/airflow/{PACKAGE_VERSION}";,
- "airflow_version": PACKAGE_VERSION,
+ "base_url":
f"https://downloads.apache.org/airflow/airflow-ctl/{PACKAGE_VERSION}";,
+ "closer_lua_url":
f"https://www.apache.org/dyn/closer.lua/airflow/airflow-ctl/{PACKAGE_VERSION}";,
+ "airflowctl_version": PACKAGE_VERSION,
},
}
+# Use for generate rst_epilog and other post-generation substitutions
+global_substitutions = {
+ "version": PACKAGE_VERSION,
+ "experimental": "This is an :ref:`experimental feature <experimental>`.",
+}
+
# -- Options for sphinx.ext.autodoc
--------------------------------------------
# See: https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html
diff --git a/airflow-ctl/docs/installation/installing-from-sources.rst
b/airflow-ctl/docs/installation/installing-from-sources.rst
index ad1e616873d..2d2f9a421f5 100644
--- a/airflow-ctl/docs/installation/installing-from-sources.rst
+++ b/airflow-ctl/docs/installation/installing-from-sources.rst
@@ -23,9 +23,8 @@ Released packages
.. jinja:: official_download_page
- This page describes downloading and verifying Airflow® version
- ``{{ airflow_version }}`` using officially released packages.
- You can also install ``Apache airflowctl`` - as most Python packages - via
:doc:`PyPI <installing-from-pypi>`.
+ This page describes downloading and verifying Airflow Ctl version
``|version|`` using officially released packages.
+ You can also install ``airflowctl`` - as most Python packages - via
:doc:`PyPI <installing-from-pypi>`.
You can choose different version of Airflow by selecting a different
version from the drop-down at
the top-left of the page.
@@ -34,22 +33,13 @@ can use if you want to verify the origin of the packages
and want to verify chec
the packages. The packages are available via the
`Official Apache Software Foundations Downloads <https://dlcdn.apache.org/>`_
-As of version 2.8 Airflow follows PEP 517/518 and uses ``pyproject.toml`` file
to define build dependencies
-and build process and it requires relatively modern versions of packaging
tools to get airflow built from
-local sources or ``sdist`` packages, as PEP 517 compliant build hooks are used
to determine dynamic build
-dependencies. In case of ``pip`` it means that at least version 22.1.0 is
needed (released at the beginning of
-2022) to build or install Airflow from sources. This does not affect the
ability of installing Airflow from
-released wheel packages.
-
-The |version| downloads of airflowctl are available at:
+The {{ airflowctl_version }} downloads of Airflow Ctl are available at:
.. jinja:: official_download_page
- * `Sources package for airflow <{{ closer_lua_url }}/apache-airflow-ctl-{{
airflowctl_version }}-source.tar.gz>`__ (`asc <{{ base_url
}}/apache-airflow-ctl-{{ airflowctl_version }}-source.tar.gz.asc>`__, `sha512
<{{ base_url }}/apache-airflow-ctl-{{ airflowctl_version
}}-source.tar.gz.sha512>`__)
- * `Sdist package for airflow meta package <{{ closer_lua_url
}}/apache-airflow-ctl-{{ airflowctl_version }}.tar.gz>`__ (`asc <{{ base_url
}}/apache-airflow-ctl-{{ airflowctl_version }}.tar.gz.asc>`__, `sha512 <{{
base_url }}/apache-airflow-ctl-{{ airflowctl_version }}.tar.gz.sha512>`__)
- * `Whl package for airflow meta package <{{ closer_lua_url
}}/apache_airflow_ctl-{{ airflowctl_version }}-py3-none-any.whl>`__ (`asc <{{
base_url }}/apache_airflow_ctl-{{ airflowctl_version
}}-py3-none-any.whl.asc>`__, `sha512 <{{ base_url }}/apache_airflow_ctl-{{
airflowctl_version }}-py3-none-any.whl.sha512>`__)
- * `Sdist package for airflow core package <{{ closer_lua_url
}}/apache-airflow_ctl-{{ airflowctl_version }}.tar.gz>`__ (`asc <{{ base_url
}}/apache-airflow_ctl-{{ airflowctl_version }}.tar.gz.asc>`__, `sha512 <{{
base_url }}/apache-airflow_ctl-{{ airflowctl_version }}.tar.gz.sha512>`__)
- * `Whl package for airflow core package <{{ closer_lua_url
}}/apache_airflow_ctl-{{ airflowctl_version }}-py3-none-any.whl>`__ (`asc <{{
base_url }}/apache_airflow_ctl-{{ airflowctl_version
}}-py3-none-any.whl.asc>`__, `sha512 <{{ base_url }}/apache_airflow_ctl-{{
airflowctl_version }}-py3-none-any.whl.sha512>`__)
+ * `Sources package for airflow-ctl: <{{ closer_lua_url
}}/apache_airflow_ctl-{{ airflowctl_version }}-source.tar.gz>`__ (`asc <{{
base_url }}/apache_airflow_ctl-{{ airflowctl_version }}-source.tar.gz.asc>`__,
`sha512 <{{ base_url }}/apache_airflow_ctl-{{ airflowctl_version
}}-source.tar.gz.sha512>`__)
+ * `Sdist package for airflow-ctl distributions <{{ closer_lua_url
}}/apache_airflow_ctl-{{ airflowctl_version }}.tar.gz>`__ (`asc <{{ base_url
}}/apache_airflow_ctl-{{ airflowctl_version }}.tar.gz.asc>`__, `sha512 <{{
base_url }}/apache_airflow_ctl-{{ airflowctl_version }}.tar.gz.sha512>`__)
+ * `Whl package for airflow-ctl distribution <{{ closer_lua_url
}}/apache_airflow_ctl-{{ airflowctl_version }}-py3-none-any.whl>`__ (`asc <{{
base_url }}/apache_airflow_ctl-{{ airflowctl_version
}}-py3-none-any.whl.asc>`__, `sha512 <{{ base_url }}/apache_airflow_ctl-{{
airflowctl_version }}-py3-none-any.whl.sha512>`__)
If you want to install from the source code, you can download from the sources
link above, it will contain
a ``INSTALL`` file containing details on how you can build and install
airflowctl.
diff --git a/dev/README_RELEASE_AIRFLOWCTL.md b/dev/README_RELEASE_AIRFLOWCTL.md
index 8d897f5242c..609339c45a2 100644
--- a/dev/README_RELEASE_AIRFLOWCTL.md
+++ b/dev/README_RELEASE_AIRFLOWCTL.md
@@ -768,8 +768,8 @@
SOURCE_DIR="${ASF_DIST_PARENT}/asf-dist/dev/airflow/airflow-ctl"
# Create airflow-ctl folder if it does not exist
# All latest releases are kept in this one folder without version sub-folder
cd "${ASF_DIST_PARENT}/asf-dist/release/airflow"
-mkdir -pv airflow-ctl
-cd airflow-ctl
+mkdir -pv airflow-ctl/${VERSION}
+cd airflow-ctl/${VERSION}
# Copy your airflow-ctl with the target name to dist directory and to SVN
rm -rf "${AIRFLOW_REPO_ROOT}"/dist/*
diff --git
a/devel-common/src/sphinx_exts/docs_build/dev_index_template.html.jinja2
b/devel-common/src/sphinx_exts/docs_build/dev_index_template.html.jinja2
index d5ea9176756..4c990817d38 100644
--- a/devel-common/src/sphinx_exts/docs_build/dev_index_template.html.jinja2
+++ b/devel-common/src/sphinx_exts/docs_build/dev_index_template.html.jinja2
@@ -56,14 +56,23 @@
</div>
</div>
- <div class="row">
- <div class="col">
- <h2><a href="/docs/task-sdk/stable/index.html">Task SDK</a></h2>
- <p>
- Task-SDK interface that is used to communicate with airflow
core from other components.
- </p>
- </div>
- </div>
+ <div class="row">
+ <div class="col">
+ <h2><a
href="/docs/apache-airflow-ctl/stable/index.html"><code>apache-airflow-ctl</code></a></h2>
+ <p>
+ Apache Airflow CTL, which is remote CLI for Airflow.
+ </p>
+ </div>
+ </div>
+
+ <div class="row">
+ <div class="col">
+ <h2><a href="/docs/task-sdk/stable/index.html">Task SDK</a></h2>
+ <p>
+ Task-SDK interface that is used to communicate with airflow core
from other components.
+ </p>
+ </div>
+ </div>
<div class="row">
<div class="col-md order-md-1">
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index 5d673b7b860..3aa45ba8fed 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -365,6 +365,7 @@ csrf
CSRFProtect
css
csv
+Ctl
ctor
Ctrl
cubeName
