potiuk commented on code in PR #74556:
URL: https://github.com/apache/airflow/pull/74556#discussion_r4307960277
##########
scripts/docker/install_airflow_when_building_images.sh:
##########
@@ -35,6 +35,104 @@
# shellcheck source=scripts/docker/common.sh
. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+function has_providers_excluded_for_python() {
+ local python_minor="${AIRFLOW_PYTHON_VERSION%.*}"
+ PYTHONPATH="$( dirname "${BASH_SOURCE[0]}" )" python3 -c \
+ 'from pathlib import Path; import sys; from filter_ci_install_set
import split_members; sys.exit(not split_members(Path.cwd(), sys.argv[1])[1])' \
+ "${python_minor}"
Review Comment:
Any Python failure here (import error, the working directory not being the
workspace root, a malformed `pyproject.toml`, `ValueError` from
`find_workspace_members`) exits 1, the same as "no excluded providers". The
build then silently takes the plain `uv sync --all-packages` path and installs
exactly what this PR is meant to avoid. Distinct exit codes (e.g. 0 / 10 for
yes / no), with the build failing on anything else, would close that.
##########
scripts/docker/filter_ci_install_set.py:
##########
@@ -0,0 +1,385 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Compute the CI image install set for one Python minor without excluded
workspace members.
+
+``uv sync --all-packages`` installs every workspace member, including the dev
groups of providers
+excluded for a Python minor (``Requires-Python`` carries ``!=<minor>``). Their
third-party
+dependencies would still be installed, and some do not build for the new
interpreter.
+
+The helper resolves the requirements of the members that are not excluded, not
the lock's export
+graph. Each kept member becomes a local ``file://`` requirement, and the
requirements of each kept
+member's ``dev`` group (the root's ``ci-image`` group for the root) are added
verbatim, with
+excluded members removed. ``uv pip compile`` resolves that input for the
target interpreter and
+platform, so markers apply per edge. The lock's pins are passed as
constraints, so every version
+matches the lock. Output is the third-party pins and the editable member
directories. With
+``--hashes``, each pin carries the lock export's ``--hash`` options so the
install can be run
+under ``--require-hashes``, which is the check ``uv sync --frozen`` would have
done.
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+import subprocess
+import sys
+import tomllib
+from glob import glob
+from pathlib import Path
+from typing import Any
+from urllib.parse import unquote, urlparse
+
+MEMBER_DEFAULT_GROUP = "dev"
+ROOT_CI_GROUP = "ci-image"
+MINOR_VERSION = re.compile(r"\d+\.\d+")
+REQUIREMENT_NAME = re.compile(r"\s*([A-Za-z0-9][A-Za-z0-9._-]*)")
+RESOLVED_MEMBER = re.compile(r"^([^\s@]+)\s*@\s*(file://\S+)$")
+PINNED_NAME =
re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)(?:\[[^\]]*\])?\s*(?:==|@)")
+PINNED_VERSION = re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)==([^\s;]+)")
+HASH_OPTION = re.compile(r"--hash=[A-Za-z0-9_]+:[A-Fa-f0-9]+")
+GLOB_CHARACTERS = frozenset("*?[")
+PYTHON_PLATFORMS = {"x86_64": "x86_64-manylinux_2_28", "aarch64":
"aarch64-manylinux_2_28"}
+
+
+def normalize_name(name: str) -> str:
+ return re.sub(r"[-_.]+", "-", name).lower()
+
+
+def requirement_name(entry: str) -> str:
+ match = REQUIREMENT_NAME.match(entry)
+ if not match:
+ raise ValueError(f"Cannot read a requirement name from {entry!r}")
+ return normalize_name(match.group(1))
+
+
+def is_python_minor_excluded(requires_python: str, minor: str) -> bool:
+ # The lookahead keeps longer versions such as !=3.150 or !=3.15.1 from
matching a whole-minor exclusion.
+ return re.search(rf"!=\s*{re.escape(minor)}(\.\*)?(?![\d.])",
requires_python) is not None
Review Comment:
This reads `requires-python` with a regex over `!=` clauses only, so upper
bounds (`<3.14`), `~=` and other specifiers are ignored: a member that is
incompatible that way is kept, and the resolve or install then fails on it. It
also departs from PEP 440, where `!=3.15` excludes only 3.15.0 (`!=3.15.*` is
the whole minor).
`packaging.specifiers.SpecifierSet(requires_python).contains(<full interpreter
version>)` would handle every form, if `packaging` is available at that point
of the build.
##########
scripts/docker/filter_ci_install_set.py:
##########
@@ -0,0 +1,385 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Compute the CI image install set for one Python minor without excluded
workspace members.
+
+``uv sync --all-packages`` installs every workspace member, including the dev
groups of providers
+excluded for a Python minor (``Requires-Python`` carries ``!=<minor>``). Their
third-party
+dependencies would still be installed, and some do not build for the new
interpreter.
+
+The helper resolves the requirements of the members that are not excluded, not
the lock's export
+graph. Each kept member becomes a local ``file://`` requirement, and the
requirements of each kept
+member's ``dev`` group (the root's ``ci-image`` group for the root) are added
verbatim, with
+excluded members removed. ``uv pip compile`` resolves that input for the
target interpreter and
+platform, so markers apply per edge. The lock's pins are passed as
constraints, so every version
+matches the lock. Output is the third-party pins and the editable member
directories. With
+``--hashes``, each pin carries the lock export's ``--hash`` options so the
install can be run
+under ``--require-hashes``, which is the check ``uv sync --frozen`` would have
done.
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+import subprocess
+import sys
+import tomllib
+from glob import glob
+from pathlib import Path
+from typing import Any
+from urllib.parse import unquote, urlparse
+
+MEMBER_DEFAULT_GROUP = "dev"
+ROOT_CI_GROUP = "ci-image"
+MINOR_VERSION = re.compile(r"\d+\.\d+")
+REQUIREMENT_NAME = re.compile(r"\s*([A-Za-z0-9][A-Za-z0-9._-]*)")
+RESOLVED_MEMBER = re.compile(r"^([^\s@]+)\s*@\s*(file://\S+)$")
+PINNED_NAME =
re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)(?:\[[^\]]*\])?\s*(?:==|@)")
+PINNED_VERSION = re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)==([^\s;]+)")
+HASH_OPTION = re.compile(r"--hash=[A-Za-z0-9_]+:[A-Fa-f0-9]+")
+GLOB_CHARACTERS = frozenset("*?[")
+PYTHON_PLATFORMS = {"x86_64": "x86_64-manylinux_2_28", "aarch64":
"aarch64-manylinux_2_28"}
+
+
+def normalize_name(name: str) -> str:
+ return re.sub(r"[-_.]+", "-", name).lower()
+
+
+def requirement_name(entry: str) -> str:
+ match = REQUIREMENT_NAME.match(entry)
+ if not match:
+ raise ValueError(f"Cannot read a requirement name from {entry!r}")
+ return normalize_name(match.group(1))
+
+
+def is_python_minor_excluded(requires_python: str, minor: str) -> bool:
+ # The lookahead keeps longer versions such as !=3.150 or !=3.15.1 from
matching a whole-minor exclusion.
+ return re.search(rf"!=\s*{re.escape(minor)}(\.\*)?(?![\d.])",
requires_python) is not None
+
+
+def load_manifest(directory: Path) -> dict[str, Any]:
+ with (directory / "pyproject.toml").open("rb") as manifest:
+ return tomllib.load(manifest)
+
+
+def has_glob_characters(pattern: str) -> bool:
+ return any(character in GLOB_CHARACTERS for character in pattern)
+
+
+def find_workspace_members(root: Path, patterns: list[str]) -> list[Path]:
+ members: list[Path] = []
+ for pattern in patterns:
+ matches = sorted(glob(str(root / pattern)))
+ if not matches and not has_glob_characters(pattern):
+ raise ValueError(f"Workspace member {pattern!r} does not exist
under {root}")
+ for match in matches:
+ candidate = Path(match)
+ if (candidate / "pyproject.toml").is_file():
+ members.append(candidate)
+ elif not has_glob_characters(pattern):
+ raise ValueError(f"Workspace member {pattern!r} has no
pyproject.toml")
+ return members
+
+
+def expand_group(groups: dict[str, list[Any]], group: str, seen: set[str] |
None = None) -> list[str]:
+ """Return the raw requirement strings of a dependency group, following
``include-group`` entries."""
+ if seen is None:
+ seen = set()
+ if group in seen:
+ return []
+ seen.add(group)
+ entries: list[str] = []
+ for entry in groups.get(group, []):
+ if isinstance(entry, dict):
+ entries.extend(expand_group(groups, entry["include-group"], seen))
+ else:
+ entries.append(entry)
+ return entries
+
+
+def split_members(root: Path, minor: str) -> tuple[dict[str, Path], set[str]]:
+ """Return the members kept for installation (name -> directory) and the
excluded member names."""
+ patterns = load_manifest(root)["tool"]["uv"]["workspace"]["members"]
+ kept: dict[str, Path] = {}
+ excluded: set[str] = set()
+ for member in find_workspace_members(root, patterns):
+ project = load_manifest(member).get("project", {})
+ name = normalize_name(project.get("name", member.name))
+ if is_python_minor_excluded(project.get("requires-python", ""), minor):
+ excluded.add(name)
+ else:
+ kept[name] = member
+ return kept, excluded
Review Comment:
This applies to every workspace member, not only providers. The root,
`airflow-core` and `task-sdk` declare `requires-python = ">=3.11,!=3.15"`, so
on a 3.15 interpreter they land in `excluded` too, and the install continues
without Airflow itself. If a non-provider member is ever excluded, failing the
build with a clear message would be safer than continuing.
##########
scripts/docker/filter_ci_install_set.py:
##########
@@ -0,0 +1,385 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Compute the CI image install set for one Python minor without excluded
workspace members.
+
+``uv sync --all-packages`` installs every workspace member, including the dev
groups of providers
+excluded for a Python minor (``Requires-Python`` carries ``!=<minor>``). Their
third-party
+dependencies would still be installed, and some do not build for the new
interpreter.
+
+The helper resolves the requirements of the members that are not excluded, not
the lock's export
+graph. Each kept member becomes a local ``file://`` requirement, and the
requirements of each kept
+member's ``dev`` group (the root's ``ci-image`` group for the root) are added
verbatim, with
+excluded members removed. ``uv pip compile`` resolves that input for the
target interpreter and
+platform, so markers apply per edge. The lock's pins are passed as
constraints, so every version
+matches the lock. Output is the third-party pins and the editable member
directories. With
+``--hashes``, each pin carries the lock export's ``--hash`` options so the
install can be run
+under ``--require-hashes``, which is the check ``uv sync --frozen`` would have
done.
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+import subprocess
+import sys
+import tomllib
+from glob import glob
+from pathlib import Path
+from typing import Any
+from urllib.parse import unquote, urlparse
+
+MEMBER_DEFAULT_GROUP = "dev"
+ROOT_CI_GROUP = "ci-image"
+MINOR_VERSION = re.compile(r"\d+\.\d+")
+REQUIREMENT_NAME = re.compile(r"\s*([A-Za-z0-9][A-Za-z0-9._-]*)")
+RESOLVED_MEMBER = re.compile(r"^([^\s@]+)\s*@\s*(file://\S+)$")
+PINNED_NAME =
re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)(?:\[[^\]]*\])?\s*(?:==|@)")
+PINNED_VERSION = re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)==([^\s;]+)")
+HASH_OPTION = re.compile(r"--hash=[A-Za-z0-9_]+:[A-Fa-f0-9]+")
+GLOB_CHARACTERS = frozenset("*?[")
+PYTHON_PLATFORMS = {"x86_64": "x86_64-manylinux_2_28", "aarch64":
"aarch64-manylinux_2_28"}
+
+
+def normalize_name(name: str) -> str:
+ return re.sub(r"[-_.]+", "-", name).lower()
+
+
+def requirement_name(entry: str) -> str:
Review Comment:
Per AGENTS.md, functions should be named with action verbs; noun-only names
read as attributes. `extract_requirement_name` here, and `get_member_path` for
`member_path` below.
##########
scripts/docker/install_airflow_when_building_images.sh:
##########
@@ -35,6 +35,104 @@
# shellcheck source=scripts/docker/common.sh
. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+function has_providers_excluded_for_python() {
+ local python_minor="${AIRFLOW_PYTHON_VERSION%.*}"
+ PYTHONPATH="$( dirname "${BASH_SOURCE[0]}" )" python3 -c \
+ 'from pathlib import Path; import sys; from filter_ci_install_set
import split_members; sys.exit(not split_members(Path.cwd(), sys.argv[1])[1])' \
+ "${python_minor}"
+}
+
+# uv sync --all-packages installs every workspace member, including providers
excluded for this
+# Python minor and the third-party packages only they need. Resolve the
members that are not excluded
+# for this interpreter, with the lock's pins as constraints, and install
exactly that result.
+# Resolves the lock into the install set for this interpreter. This is the
only step that can fail
+# recoverably - a dependency missing from uv.lock stops the helper - so the
caller tests it, and that
+# suppresses errexit here: each command has to report its own failure.
+function prepare_filtered_install_set() {
+ local status
+ # Two exports of the same lock: the plain one is the resolver's
constraints file (uv rejects hashes
+ # there), the hashed one supplies the --hash options that keep `uv sync
--frozen`'s sha256 check.
+ uv export --frozen --all-packages --group ci-image --no-hashes
--no-emit-workspace \
+ --output-file "${work_dir}/pins.txt" || { status=$?; set +x; return
"${status}"; }
+ uv export --frozen --all-packages --group ci-image --no-emit-workspace \
+ --output-file "${work_dir}/hashed-pins.txt" || { status=$?; set +x;
return "${status}"; }
+ python3 "$( dirname "${BASH_SOURCE[0]}" )/filter_ci_install_set.py" --root
"${PWD}" \
+ --constraints "${work_dir}/pins.txt" "${helper_hash_args[@]}" \
+ --python-minor "${python_minor}" --platform "${platform_arch}" \
+ --workdir "${work_dir}" --output "${work_dir}/requirements.txt" \
+ --editables "${work_dir}/editables.txt" || { status=$?; set +x; return
"${status}"; }
+}
+
+# Pass "allow-relock" to let a stale lock be re-resolved instead of failing
the build.
+function install_workspace_without_excluded_providers() {
+ local allow_relock="${1:-}"
+ local python_minor="${AIRFLOW_PYTHON_VERSION%.*}"
+ local work_dir="/tmp/ci-install-set"
+ local python_bin
+ local platform_arch
+ local status=0
+ # Hash enforcement travels as arrays so a later caller can turn it off
coherently: uv verifies a
+ # hash whenever the requirements file carries one, so skipping
--require-hashes alone is not enough.
+ # --no-config rides with --require-hashes: the install runs from the
workspace root, and uv would
+ # otherwise apply the root's constraint-dependencies, which
--require-hashes rejects for not being
+ # pinned with "==". Nothing is lost - these are exact pins installed with
--no-deps.
+ local -a hash_args=(--require-hashes --no-config)
Review Comment:
`--no-config` drops all root `[tool.uv]` settings, not just
`constraint-dependencies`: also `no-build-isolation-package` (`sphinx-redoc`)
and the `exclude-newer` / `exclude-newer-package` policy. A pinned package
built from an sdist then gets build isolation, unlike under `uv sync --frozen`,
so "Nothing is lost" doesn't quite hold.
##########
scripts/docker/install_airflow_when_building_images.sh:
##########
@@ -35,6 +35,104 @@
# shellcheck source=scripts/docker/common.sh
. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+function has_providers_excluded_for_python() {
+ local python_minor="${AIRFLOW_PYTHON_VERSION%.*}"
+ PYTHONPATH="$( dirname "${BASH_SOURCE[0]}" )" python3 -c \
+ 'from pathlib import Path; import sys; from filter_ci_install_set
import split_members; sys.exit(not split_members(Path.cwd(), sys.argv[1])[1])' \
+ "${python_minor}"
+}
+
+# uv sync --all-packages installs every workspace member, including providers
excluded for this
+# Python minor and the third-party packages only they need. Resolve the
members that are not excluded
+# for this interpreter, with the lock's pins as constraints, and install
exactly that result.
+# Resolves the lock into the install set for this interpreter. This is the
only step that can fail
+# recoverably - a dependency missing from uv.lock stops the helper - so the
caller tests it, and that
+# suppresses errexit here: each command has to report its own failure.
+function prepare_filtered_install_set() {
+ local status
+ # Two exports of the same lock: the plain one is the resolver's
constraints file (uv rejects hashes
+ # there), the hashed one supplies the --hash options that keep `uv sync
--frozen`'s sha256 check.
+ uv export --frozen --all-packages --group ci-image --no-hashes
--no-emit-workspace \
+ --output-file "${work_dir}/pins.txt" || { status=$?; set +x; return
"${status}"; }
+ uv export --frozen --all-packages --group ci-image --no-emit-workspace \
+ --output-file "${work_dir}/hashed-pins.txt" || { status=$?; set +x;
return "${status}"; }
+ python3 "$( dirname "${BASH_SOURCE[0]}" )/filter_ci_install_set.py" --root
"${PWD}" \
+ --constraints "${work_dir}/pins.txt" "${helper_hash_args[@]}" \
+ --python-minor "${python_minor}" --platform "${platform_arch}" \
+ --workdir "${work_dir}" --output "${work_dir}/requirements.txt" \
+ --editables "${work_dir}/editables.txt" || { status=$?; set +x; return
"${status}"; }
+}
+
+# Pass "allow-relock" to let a stale lock be re-resolved instead of failing
the build.
+function install_workspace_without_excluded_providers() {
+ local allow_relock="${1:-}"
+ local python_minor="${AIRFLOW_PYTHON_VERSION%.*}"
+ local work_dir="/tmp/ci-install-set"
+ local python_bin
+ local platform_arch
+ local status=0
+ # Hash enforcement travels as arrays so a later caller can turn it off
coherently: uv verifies a
+ # hash whenever the requirements file carries one, so skipping
--require-hashes alone is not enough.
+ # --no-config rides with --require-hashes: the install runs from the
workspace root, and uv would
+ # otherwise apply the root's constraint-dependencies, which
--require-hashes rejects for not being
+ # pinned with "==". Nothing is lost - these are exact pins installed with
--no-deps.
+ local -a hash_args=(--require-hashes --no-config)
+ local -a helper_hash_args=(--hashes "${work_dir}/hashed-pins.txt")
+ local -a member_dirs=()
+ local -a editable_args=()
+ local member_dir
+ mkdir -p "${work_dir}"
+ # The PROD build stage puts the base interpreter (/usr/python/bin) ahead
of the venv on PATH, so
+ # `command -v python3` would point the install at the base interpreter
instead of ${VIRTUAL_ENV}.
+ # The CI image sets no VIRTUAL_ENV and keeps resolving python3 from PATH.
+ python_bin="${VIRTUAL_ENV:+${VIRTUAL_ENV}/bin/python}"
+ python_bin="${python_bin:-$(command -v python3)}"
+ case "$(uname -m)" in
+ x86_64 | aarch64) platform_arch="$(uname -m)" ;;
+ *)
+ echo "Unsupported architecture for the filtered install: $(uname
-m)"
+ exit 1
+ ;;
+ esac
+ echo
+ echo "${COLOR_BLUE}Installing the workspace without providers excluded for
Python ${python_minor}.${COLOR_RESET}"
+ echo
+ set -x
+ prepare_filtered_install_set || status=$?
+ set +x
+ if [[ ${status} -ne 0 ]]; then
+ if [[ ${allow_relock} != "allow-relock" ||
${AIRFLOW_FALLBACK_NO_CONSTRAINTS_INSTALLATION} != "true" ]]; then
+ echo
+ echo "${COLOR_RED}Failing because the filtered installation failed
and fallback is disabled.${COLOR_RESET}"
+ echo
+ exit "${status}"
+ fi
+ echo
+ echo "${COLOR_YELLOW}Likely pyproject.toml has new dependencies not
reflected in uv.lock.${COLOR_RESET}"
+ echo
+ echo "${COLOR_BLUE}Falling back to re-resolving dependencies (uv lock,
then filtered install).${COLOR_RESET}"
+ echo
+ set -x
+ uv lock --no-python-downloads --no-managed-python
+ prepare_filtered_install_set
+ set +x
+ fi
+ # Nothing below may fail silently, and nothing has to say so: no caller
tests this function, so
+ # errexit is in force from here on.
+ set -x
+ uv pip install --python "${python_bin}" --no-deps --no-python-downloads
--no-managed-python \
+ "${hash_args[@]}" --no-binary lxml --no-binary xmlsec \
+ --requirement "${work_dir}/requirements.txt"
Review Comment:
Unlike `uv sync`, `uv pip install --no-deps` doesn't remove packages that
are no longer in the set. In cached or incremental image builds, dependencies
of the excluded providers that were installed earlier stay in the image. It's
also worth checking that the target interpreter matches where `uv sync`
installs in the CI image (`UV_PROJECT_ENVIRONMENT=/usr/local`).
##########
scripts/docker/filter_ci_install_set.py:
##########
@@ -0,0 +1,385 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Compute the CI image install set for one Python minor without excluded
workspace members.
+
+``uv sync --all-packages`` installs every workspace member, including the dev
groups of providers
+excluded for a Python minor (``Requires-Python`` carries ``!=<minor>``). Their
third-party
+dependencies would still be installed, and some do not build for the new
interpreter.
+
+The helper resolves the requirements of the members that are not excluded, not
the lock's export
+graph. Each kept member becomes a local ``file://`` requirement, and the
requirements of each kept
+member's ``dev`` group (the root's ``ci-image`` group for the root) are added
verbatim, with
+excluded members removed. ``uv pip compile`` resolves that input for the
target interpreter and
+platform, so markers apply per edge. The lock's pins are passed as
constraints, so every version
+matches the lock. Output is the third-party pins and the editable member
directories. With
+``--hashes``, each pin carries the lock export's ``--hash`` options so the
install can be run
+under ``--require-hashes``, which is the check ``uv sync --frozen`` would have
done.
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+import subprocess
+import sys
+import tomllib
+from glob import glob
+from pathlib import Path
+from typing import Any
+from urllib.parse import unquote, urlparse
+
+MEMBER_DEFAULT_GROUP = "dev"
+ROOT_CI_GROUP = "ci-image"
+MINOR_VERSION = re.compile(r"\d+\.\d+")
+REQUIREMENT_NAME = re.compile(r"\s*([A-Za-z0-9][A-Za-z0-9._-]*)")
+RESOLVED_MEMBER = re.compile(r"^([^\s@]+)\s*@\s*(file://\S+)$")
+PINNED_NAME =
re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)(?:\[[^\]]*\])?\s*(?:==|@)")
+PINNED_VERSION = re.compile(r"^([A-Za-z0-9][A-Za-z0-9._-]*)==([^\s;]+)")
+HASH_OPTION = re.compile(r"--hash=[A-Za-z0-9_]+:[A-Fa-f0-9]+")
+GLOB_CHARACTERS = frozenset("*?[")
+PYTHON_PLATFORMS = {"x86_64": "x86_64-manylinux_2_28", "aarch64":
"aarch64-manylinux_2_28"}
Review Comment:
`manylinux_2_28` is older than the image's glibc (2.36 on bookworm). A
locked package that only ships `manylinux_2_34+` wheels installs fine with `uv
sync --frozen` but has no compatible distribution for this resolve, which fails
the build. `--python-version <minor>` also evaluates `python_full_version`
markers against X.Y.0 rather than the real patch release.
##########
scripts/docker/install_airflow_when_building_images.sh:
##########
@@ -35,6 +35,104 @@
# shellcheck source=scripts/docker/common.sh
. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+function has_providers_excluded_for_python() {
+ local python_minor="${AIRFLOW_PYTHON_VERSION%.*}"
+ PYTHONPATH="$( dirname "${BASH_SOURCE[0]}" )" python3 -c \
+ 'from pathlib import Path; import sys; from filter_ci_install_set
import split_members; sys.exit(not split_members(Path.cwd(), sys.argv[1])[1])' \
+ "${python_minor}"
+}
+
+# uv sync --all-packages installs every workspace member, including providers
excluded for this
+# Python minor and the third-party packages only they need. Resolve the
members that are not excluded
+# for this interpreter, with the lock's pins as constraints, and install
exactly that result.
+# Resolves the lock into the install set for this interpreter. This is the
only step that can fail
+# recoverably - a dependency missing from uv.lock stops the helper - so the
caller tests it, and that
+# suppresses errexit here: each command has to report its own failure.
+function prepare_filtered_install_set() {
+ local status
+ # Two exports of the same lock: the plain one is the resolver's
constraints file (uv rejects hashes
+ # there), the hashed one supplies the --hash options that keep `uv sync
--frozen`'s sha256 check.
+ uv export --frozen --all-packages --group ci-image --no-hashes
--no-emit-workspace \
+ --output-file "${work_dir}/pins.txt" || { status=$?; set +x; return
"${status}"; }
+ uv export --frozen --all-packages --group ci-image --no-emit-workspace \
+ --output-file "${work_dir}/hashed-pins.txt" || { status=$?; set +x;
return "${status}"; }
Review Comment:
The same lock is exported twice per build (with and without hashes). A
single hashed export, with the `--hash` options stripped for the constraints
file, would do. Also, `pyproject.toml` of every member is parsed about three
times (the probe, `split_members`, `build_requirement_input`); `split_members`
could return the loaded manifests for reuse.
##########
scripts/docker/install_airflow_when_building_images.sh:
##########
@@ -35,6 +35,104 @@
# shellcheck source=scripts/docker/common.sh
. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+function has_providers_excluded_for_python() {
+ local python_minor="${AIRFLOW_PYTHON_VERSION%.*}"
+ PYTHONPATH="$( dirname "${BASH_SOURCE[0]}" )" python3 -c \
+ 'from pathlib import Path; import sys; from filter_ci_install_set
import split_members; sys.exit(not split_members(Path.cwd(), sys.argv[1])[1])' \
+ "${python_minor}"
+}
+
+# uv sync --all-packages installs every workspace member, including providers
excluded for this
+# Python minor and the third-party packages only they need. Resolve the
members that are not excluded
+# for this interpreter, with the lock's pins as constraints, and install
exactly that result.
+# Resolves the lock into the install set for this interpreter. This is the
only step that can fail
+# recoverably - a dependency missing from uv.lock stops the helper - so the
caller tests it, and that
+# suppresses errexit here: each command has to report its own failure.
Review Comment:
These two comment blocks overlap. The first (lines 45-47) describes
`install_workspace_without_excluded_providers`, and the second partly repeats
it. Moving the first above that function and keeping one short block here would
avoid misleading readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
