Dev-iL commented on code in PR #74556: URL: https://github.com/apache/airflow/pull/74556#discussion_r4512677607
########## scripts/docker/filter_ci_install_set.py: ########## @@ -0,0 +1,146 @@ +#!/usr/bin/env python +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Select the workspace members the CI image installs for one Python minor. + +``uv sync --all-packages`` installs every workspace member, including providers excluded for this +Python minor, and the third-party packages only those providers reach - some of which have no wheels +for the new interpreter. Listing the members that are kept lets the installer export the lock for +exactly those: ``uv export --package <member>`` emits the lock's own closure, already carrying the +hashes and the environment markers that encode each provider's exclusions, so the install filters +itself on the target interpreter. No second resolution runs, so nothing can drift from the lock. + +Exclusions are read from ``generated/provider_dependencies.json``, which the provider tooling +regenerates from each ``provider.yaml``. Only providers appear there, so a core distribution can +never drop out of the install set by accident. +""" + +from __future__ import annotations + +import argparse +import json +import sys +import tomllib +from glob import glob +from pathlib import Path + +PROVIDER_DEPENDENCIES_PATH = Path("generated") / "provider_dependencies.json" +GLOB_CHARACTERS = frozenset("*?[") +# Exit codes the shell probe distinguishes: anything else means the probe itself failed. +EXIT_HAS_EXCLUDED_MEMBERS = 0 +EXIT_NO_EXCLUDED_MEMBERS = 10 + + +def has_glob_characters(pattern: str) -> bool: + return any(character in GLOB_CHARACTERS for character in pattern) + + +def find_workspace_members(root: Path, patterns: list[str]) -> list[Path]: + """Expand the ``[tool.uv.workspace] members`` patterns into directories holding a manifest.""" + members: list[Path] = [] + for pattern in patterns: + if has_glob_characters(pattern): + matches = [Path(match) for match in glob(str(root / pattern))] + members.extend(match for match in matches if (match / "pyproject.toml").is_file()) + continue + member = root / pattern + if not member.is_dir(): + raise ValueError(f"Workspace member {pattern!r} does not exist under {root}") + if not (member / "pyproject.toml").is_file(): + raise ValueError(f"Workspace member {pattern!r} has no pyproject.toml") + members.append(member) + return sorted(set(members)) + + +def load_manifest(directory: Path) -> dict: + return tomllib.loads((directory / "pyproject.toml").read_text(encoding="utf-8")) + + +def read_distribution_name(directory: Path) -> str: + name = load_manifest(directory).get("project", {}).get("name") + if not name: + raise ValueError(f"No project.name in {directory / 'pyproject.toml'}") + return name + + +def build_provider_distribution_name(provider_id: str) -> str: + """Turn a provider id from the generated metadata (``apache.kafka``) into its distribution name.""" + return "apache-airflow-providers-" + provider_id.replace(".", "-") + + +def read_excluded_distributions(root: Path, minor: str) -> set[str]: + """Return the distribution names of providers excluded for ``minor``.""" + metadata = json.loads((root / PROVIDER_DEPENDENCIES_PATH).read_text(encoding="utf-8")) Review Comment: Agreed. Nothing in the image build reads or copies `generated/provider_dependencies.json` any more. The exclusions now come from each provider's git-tracked `pyproject.toml`, whose `requires-python` Breeze writes from `excluded-python-versions`. The helper parses it strictly: it accepts only the two clauses Breeze writes and stops the build on anything else. A plain `docker build` of the 3.14 PROD image from a fresh worktree, with no JSON, now builds through the filtered path. <details> <summary>Explanation</summary> One correction on which paths broke. Breeze already regenerates the file: every Breeze command except `--help` calls `generate_provider_dependencies_if_needed()` at startup (`dev/breeze/src/airflow_breeze/commands/main_command.py`). That rewrites the file when it is missing, or when the provider manifests changed since it was written. With the file moved aside, `breeze prod-image build --python 3.14` logged `Regenerating provider dependencies file` and built. What did break is a plain `docker build` of either Dockerfile, on a checkout with no copy of the file or with one older than the last provider change. Breeze writes each provider's `requires-python` as `>=3.11` plus one `!=<entry>.*` per `excluded-python-versions` entry (`dev/breeze/src/airflow_breeze/utils/packages.py`), and the `update-providers-build-files` hook keeps it in sync. The file is already in every build context, including the manifests-only stage, and parses with `tomllib`, so the probe still runs on a bare interpreter. On the two routes you named: - Generating the JSON: Breeze already does it, and the plain `docker build` would still be broken. - Reading `provider.yaml`: at that point of the build only pip and uv are installed, so there is no PyYAML to read it with. `provider.yaml` isn't in the manifests stage either. This does mean `requires-python` is parsed again, which I said earlier wouldn't happen. This time the parsing is strict. The helper understands exactly `>=<version>` and `!=<version>.*`, evaluated as PEP 440 does against the interpreter's full release, so a patch entry such as `!=3.13.1.*` matches by prefix. Any other clause stops the build, and the error names the `pyproject.toml` and the clause. That covers `<3.14`, `~=`, and `!=3.15` without the wildcard, so the earlier blind spot can't come back silently. Only provider members are evaluated, so the root, `airflow-core` and `task-sdk` still can't drop out. </details> <details> <summary>Tests</summary> - `test_is_python_version_allowed_agrees_with_pep_440` checks the evaluator against `packaging.specifiers`. - `test_is_python_version_allowed_rejects_clauses_breeze_does_not_write` covers the rejected clauses. - `test_split_members_names_the_manifest_and_clause_it_cannot_evaluate` and `test_split_members_names_a_manifest_it_cannot_read` cover the error messages. - `test_source_install_follows_the_real_helper` runs the real helper under the installer with only `uv` stubbed, so the probe's exit codes and the helper's options can't drift apart from the shell. - The fixture holds only manifests, with no `generated/` directory. On the real workspace, the helper's exclusions for 3.11 to 3.15 match what the `provider.yaml` files declare: only `apache-airflow-providers-apache-tinkerpop`, only on 3.14. That holds both in the full tree and in a copy holding only the `pyproject.toml` files and `uv.lock`. The plain `docker build` of the 3.14 PROD image, from a fresh worktree with no `generated/provider_dependencies.json` (run before the latest rebase), logged: ``` Providers excluded for Python 3.14.10: apache-airflow-providers-apache-tinkerpop; kept members: 140 + uv lock --check --no-python-downloads --no-managed-python + uv pip install --python /home/airflow/.local/bin/python --exact --no-deps ... --require-hashes ... --requirement /tmp/ci-install-set/requirements.txt Resolved 777 packages + uv pip install --python /home/airflow/.local/bin/python --no-deps ... --editable /opt/airflow --editable /opt/airflow/airflow-core ... Resolved 140 packages + uv pip check --python /home/airflow/.local/bin/python Checked 917 packages All installed packages are compatible ``` </details> --- Drafted-by: Claude Code (Opus 5.5); reviewed by @Dev-iL before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
