kaxil commented on code in PR #76575:
URL: https://github.com/apache/airflow/pull/76575#discussion_r4539697368


##########
dev/breeze/src/airflow_breeze/commands/registry_commands.py:
##########
@@ -145,7 +145,28 @@ def extract_data(python: str, provider: str | None, 
allow_unreleased: bool):
         bring_compose_project_down(preserve_volumes=False, 
shell_params=shell_params)
         remove_docker_networks([f"{unique_project_name}_default"])
         fix_ownership_using_docker()
-    sys.exit(result.returncode)
+    if result.returncode:

Review Comment:
   Nothing fails if this early exit goes away: the dating step then runs on the 
old `modules.json` and returns 0, so a failed container run would publish stale 
data. Could one test have `execute_command_in_shell` return non-zero and assert 
the exit code and a single `run_command` call?



##########
dev/registry/fill_durable_versions.py:
##########
@@ -0,0 +1,172 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Date each durable module in modules.json against the provider tags that 
first shipped it.
+
+Runs on the host rather than in Breeze with the rest of the module extraction, 
because it reads
+the ``providers-<id>/<version>`` git tags and Breeze does not mount ``.git``.
+
+Usage:
+    python dev/registry/fill_durable_versions.py --modules 
dev/registry/modules.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import functools
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+from packaging.version import InvalidVersion, Version
+
+AIRFLOW_ROOT = Path(__file__).parent.parent.parent
+
+# Patterns matching the mechanism, never its registry declaration, and 
earliest match wins.

Review Comment:
   This comment is about `DURABLE_TOKENS` but now sits above 
`EXTRACTION_ONLY_FIELDS`. Swap the two blocks?



##########
dev/registry/registry_contract_models.py:
##########
@@ -146,17 +146,33 @@ class ModuleContract(BaseModel):
     provider_name: str | None = None
     supports_durable_execution: bool = False
     supports_deferrable: bool = False
+    durable_gating_provider: str | None = None

Review Comment:
   Once a publish from main puts these on S3, re-publishing any wave tagged 
before this merges fails the registry step. The publish workflow runs from the 
tag by default, so the merge uses that tag's `ModuleContract`, which is 
`extra="forbid"`. I ran `providers/2026-11-03`'s `validate_modules_catalog` 
over the live `modules.json` plus these three keys and it rejects them, and a 
failed `update-registry` also skips the docs sync. 11-03 went out twice, three 
days apart, so how do we want to handle that window?



##########
dev/registry/merge_registry_data.py:
##########
@@ -88,7 +91,7 @@ def merge(
     merged_modules.sort(key=lambda m: date_by_provider.get(m["provider_id"], 
""), reverse=True)
 
     providers_payload = validate_providers_catalog({"providers": 
merged_providers})
-    modules_payload = validate_modules_catalog({"modules": merged_modules})
+    modules_payload = validate_extracted_modules_catalog({"modules": 
merged_modules})

Review Comment:
   Could this stay on `validate_modules_catalog`? In CI, 
`fill_durable_versions.py` has already stripped the two internal fields by the 
time merge runs, so the extracted contract only drops the `extra="forbid"` 
check on what we publish, and nothing else calls `validate_modules_catalog` now.



##########
dev/registry/extract_parameters.py:
##########
@@ -566,17 +572,77 @@ def is_durable_capable(cls: type, resumable_mixin: type | 
None) -> bool:
     2. Genuinely implementing ResumableJobMixin's contract, where `execute()` 
(or a class it
     delegates to) calls `execute_resumable()`.
     """
+    return find_durable_source(cls, resumable_mixin) is not None
+
+
+def find_durable_source(cls: type, resumable_mixin: type | None) -> 
tuple[type, str] | None:
+    """Return the class a durable capability comes from and the mechanism it 
uses, or None."""
     declaring_cls = _find_marker_declaring_class(cls)
     if declaring_cls is not None and _delegates_execute_to(cls, declaring_cls, 
_MAX_DEFERRAL_WALK_DEPTH):
-        return True
+        return declaring_cls, DURABLE_MECHANISM_MARKER
 
     if resumable_mixin is None or resumable_mixin not in cls.__mro__:
-        return False
+        return None
 
     if inspect.isabstract(cls):
-        return False
+        return None
+
+    owner = _execute_chain_calls_resumable(cls, _MAX_DEFERRAL_WALK_DEPTH)
+    return (owner, DURABLE_MECHANISM_MIXIN) if owner is not None else None
 
-    return _execute_chain_calls_resumable(cls, _MAX_DEFERRAL_WALK_DEPTH)
+
+def _repo_relative_source(cls: type) -> Path | None:
+    try:
+        source_file = inspect.getsourcefile(cls)
+    except TypeError:
+        return None
+    if source_file is None:
+        return None
+    try:
+        return Path(source_file).resolve().relative_to(AIRFLOW_ROOT.resolve())
+    except ValueError:
+        return None
+
+
+def _provider_id_from_source(rel_path: Path) -> str | None:
+    parts = rel_path.parts
+    if not parts or parts[0] != "providers" or "src" not in parts:
+        return None
+    return "-".join(parts[1 : parts.index("src")])
+
+
+def get_durable_metadata(cls: type, resumable_mixin: type | None) -> dict:
+    """Build the durable-execution fields of a module entry."""
+    source = find_durable_source(cls, resumable_mixin)
+    if source is None:
+        return {
+            "supports_durable_execution": False,
+            "durable_mechanism": None,
+            "durable_gating_provider": None,
+            "durable_gating_class": None,
+            "durable_since_version": None,
+            "durable_opt_in": False,
+        }
+
+    declaring_cls, mechanism = source
+    # Gated by whoever ships the declaring class, not by `cls`'s own provider: 
EksPodOperator
+    # ships in amazon but inherits durability from KubernetesPodOperator in 
cncf-kubernetes.
+    rel_path = _repo_relative_source(declaring_cls)

Review Comment:
   Every durable fixture lives in the test file, so `_provider_id_from_source` 
gives `None` for parent and subclass alike, and passing `cls` here instead of 
`declaring_cls` stays green. Could a test patch `_repo_relative_source` so the 
parent resolves under `providers/cncf/kubernetes` and the subclass under 
`providers/amazon`, and assert the subclass gets `cncf-kubernetes`?



##########
dev/registry/tests/test_fill_durable_versions.py:
##########
@@ -0,0 +1,281 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import json
+import subprocess
+from unittest import mock
+
+import pytest
+from fill_durable_versions import (
+    DURABLE_TOKENS,
+    GitCommandError,
+    build_source_path,
+    durable_group,
+    find_first_releases,
+    get_released_tags,
+    main,
+    run_git,
+)
+
+POD_PATH = 
"providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py"
+POD_CLASS = 
"airflow.providers.cncf.kubernetes.operators.pod.KubernetesPodOperator"
+
+
[email protected](autouse=True)
+def clear_tag_cache():
+    get_released_tags.cache_clear()
+    yield
+    get_released_tags.cache_clear()
+
+
+def fake_git(tags=(), matches=(), token=None, paths=(), allow_grep=True):
+    """Stand in for run_git. The grep argv is checked, since without `-lE` git 
prints
+    `tag:path:line` and nothing gets dated."""
+
+    def run(*args, **kwargs):
+        if args[0] == "tag":
+            return list(tags)
+        if args[0] == "grep":
+            if not allow_grep:
+                raise AssertionError("grep ran even though there were no tags 
to search")
+            assert args[1] == "-lE", f"grep must list file names only, got 
{args[1]!r}"
+            if token is not None:
+                assert args[2] == token
+            assert list(args[3 : 3 + len(tags)]) == list(tags)
+            assert args[3 + len(tags)] == "--"
+            if paths:
+                assert list(args[4 + len(tags) :]) == sorted(paths)
+            assert kwargs["allowed_returncodes"] == (0, 1)
+            return list(matches)
+        raise AssertionError(f"unexpected git call: {args}")
+
+    return run
+
+
+class TestRunGit:
+    @mock.patch("fill_durable_versions.subprocess.run", autospec=True)
+    def test_unexpected_exit_code_is_raised_with_stderr(self, run):
+        run.return_value = subprocess.CompletedProcess(
+            args=[], returncode=128, stdout="", stderr="fatal: not a git 
repository"
+        )
+        with pytest.raises(GitCommandError, match="fatal: not a git 
repository"):
+            run_git("tag", "--list")
+
+    @mock.patch("fill_durable_versions.subprocess.run", autospec=True)
+    def test_allowed_exit_code_is_an_empty_answer(self, run):
+        run.return_value = subprocess.CompletedProcess(args=[], returncode=1, 
stdout="", stderr="")
+        assert run_git("grep", "-lE", "x", allowed_returncodes=(0, 1)) == []
+
+
+class TestBuildSourcePath:
+    @pytest.mark.parametrize(
+        ("provider_id", "qualified_class", "expected"),
+        [
+            ("cncf-kubernetes", POD_CLASS, POD_PATH),
+            (
+                "amazon",
+                "airflow.providers.amazon.aws.operators.glue.GlueJobOperator",
+                
"providers/amazon/src/airflow/providers/amazon/aws/operators/glue.py",
+            ),
+        ],
+    )
+    def test_multi_part_provider_id_becomes_a_directory(self, provider_id, 
qualified_class, expected):
+        assert build_source_path(provider_id, qualified_class) == expected
+
+
+class TestDurableGroup:
+    def test_marker_and_mixin_get_different_tokens(self):
+        marker = durable_group(
+            {
+                "durable_mechanism": "marker",
+                "durable_gating_provider": "cncf-kubernetes",
+                "durable_gating_class": POD_CLASS,
+            }
+        )
+        mixin = durable_group(
+            {
+                "durable_mechanism": "resumable_mixin",
+                "durable_gating_provider": "apache-livy",
+                "durable_gating_class": 
"airflow.providers.apache.livy.operators.livy.LivyOperator",
+            }
+        )
+        assert marker == ("cncf-kubernetes", DURABLE_TOKENS["marker"], 
POD_PATH)
+        assert mixin[1] == "execute_resumable"
+
+    def test_marker_token_also_catches_the_durable_parameter(self):

Review Comment:
   This only checks that both strings appear in the constant, it never runs the 
pattern. Could it run `git grep -E`, or `re.search`, against the AgentOperator 
0.1.0 signature line instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to