This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v3-3-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-3-test by this push:
new 9c18c7be40b Fix lang-SDK k8s test on release branches by using their
own SDKs (#71640)
9c18c7be40b is described below
commit 9c18c7be40b9886baea44055c7bc10f18845efb4
Author: Jarek Potiuk <[email protected]>
AuthorDate: Fri Aug 14 23:23:31 2026 +0200
Fix lang-SDK k8s test on release branches by using their own SDKs (#71640)
The Go/Java SDK sources were always fetched from upstream main, on the
assumption that a release branch either lacks them or carries a stale
copy. That copy is not stale, it is the matching one: a packed bundle
declares a dated supervisor_schema_version and the task-SDK supervisor
refuses a bundle whose version it does not know, so upstream main's SDK
cannot run against a release branch's Airflow at all.
On v3-3-test every run of the test therefore failed with "cannot find
executable bundle with usable supervisor_schema_version" as soon as main
moved to a newer schema version than the branch.
---
.../airflow_breeze/commands/kubernetes_commands.py | 52 ++++++++---------
.../tests/test_kubernetes_lang_sdk_commands.py | 65 +++++++++++-----------
kubernetes-tests/lang_sdk/README.md | 39 ++++++-------
3 files changed, 77 insertions(+), 79 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
b/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
index 84fac9d60f4..ac6ae2752a0 100644
--- a/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
@@ -32,7 +32,6 @@ from typing import Any
import click
import yaml
-from airflow_breeze.branch_defaults import AIRFLOW_BRANCH
from airflow_breeze.commands.common_options import (
option_answer,
option_debug_resources,
@@ -2502,39 +2501,34 @@ LANG_SDK_AWS_CONN_URI = (
"aws://test:test@/?region_name=us-east-1&"
"endpoint_url=http%3A%2F%2Flocalstack.airflow.svc.cluster.local%3A4566"
)
-# Runs targeting a branch other than main build the Go/Java SDKs from upstream
main, so
-# release/backport branches with stale or missing go-sdk/java-sdk copies still
test current SDK
-# sources. See kubernetes-tests/lang_sdk/README.md.
+# Only a checkout that carries no go-sdk/java-sdk of its own falls back to
these.
+# See kubernetes-tests/lang_sdk/README.md.
LANG_SDK_UPSTREAM_GIT_URL = "https://github.com/apache/airflow.git";
LANG_SDK_UPSTREAM_REF = "main"
-# The target branch for which the lang-SDK artifacts are built from this
checkout's own SDK
-# sources rather than fetched from upstream. Distinct from
LANG_SDK_UPSTREAM_REF (the ref fetched
-# from upstream) even though both are "main" today -- one names a git ref, the
other a sentinel.
-LANG_SDK_LOCAL_SOURCE_BRANCH = "main"
-
-
-def _lang_sdk_target_branch() -> str:
- """Branch this run targets: GITHUB_BASE_REF or DEFAULT_BRANCH if set, else
this checkout's."""
- return os.environ.get("GITHUB_BASE_REF") or
os.environ.get("DEFAULT_BRANCH") or AIRFLOW_BRANCH
def _lang_sdk_resolve_sdk_sources(staging: Path, output: Output | None) ->
tuple[Path, Path]:
"""Resolve the go-sdk/java-sdk trees the lang-SDK artifacts are built from.
- A run targeting main builds the checked-out branch's own SDK sources, so a
PR's SDK changes
- are what the k8s test exercises -- without this,
``java_example``/``go_example`` (harness code
- that tracks the branch) is compiled against a different SDK than the
branch it belongs to, and
- any SDK rename breaks the build. Runs targeting anything else fall back to
upstream main.
+ The checkout's own sources win whenever it has them, because they are the
ones that pair with
+ the Airflow this test deploys: a packed bundle declares a dated
``supervisor_schema_version``
+ and the task-SDK supervisor rejects a bundle whose version it does not
know, so a release
+ branch's Airflow cannot run an SDK built from a later main. Building the
checkout's copy is
+ also what makes the k8s test exercise a PR's SDK changes --
``go_example``/``java_example`` are
+ harness fixtures that track the checked-out branch, so compiling them
against a different SDK
+ breaks on any SDK rename.
+
+ Only a branch cut before ``go-sdk``/``java-sdk`` existed falls back to
upstream main.
"""
- target = _lang_sdk_target_branch()
- if target == LANG_SDK_LOCAL_SOURCE_BRANCH:
+ go_sdk, java_sdk = AIRFLOW_ROOT_PATH / "go-sdk", AIRFLOW_ROOT_PATH /
"java-sdk"
+ if go_sdk.is_dir() and java_sdk.is_dir():
get_console(output=output).print(
- f"[info]Run targets {target}: building the lang-SDK Go/Java
artifacts from this branch"
+ "[info]Building the lang-SDK Go/Java artifacts from this
checkout's own go-sdk/java-sdk"
)
- return AIRFLOW_ROOT_PATH / "go-sdk", AIRFLOW_ROOT_PATH / "java-sdk"
+ return go_sdk, java_sdk
get_console(output=output).print(
- f"[info]Run targets {target}, not {LANG_SDK_LOCAL_SOURCE_BRANCH}:
building the lang-SDK Go/Java "
- f"artifacts from upstream {LANG_SDK_UPSTREAM_REF}"
+ f"[info]This checkout has no go-sdk/java-sdk: building the lang-SDK
Go/Java artifacts from "
+ f"upstream {LANG_SDK_UPSTREAM_REF}"
)
return _lang_sdk_fetch_upstream_sdk_sources(staging, output)
@@ -2619,7 +2613,7 @@ def _lang_sdk_build_go_bundle(
workspace mirroring the repo layout with ``go_sdk_source`` at
``<workspace>/go-sdk``, letting
the unmodified directive resolve against it. The scratch go_example is
re-tidied before packing
so its go.sum reconciles to that go-sdk, which differs from the in-repo
one its committed go.sum
- was tidied against whenever the source is upstream main.
+ was tidied against whenever the source is the upstream-main fallback.
"""
go_dir = staging / "go-artifacts"
go_dir.mkdir(parents=True, exist_ok=True)
@@ -2639,8 +2633,8 @@ def _lang_sdk_build_go_bundle(
# the current dir (".") because go_example is its own module.
#
# go_example's go.sum is tidied against the in-repo go-sdk, but the bundle
is built against the
- # upstream-main go-sdk copied in above. When a branch changes go-sdk's
dependency graph those two
- # go-sdks differ, and Go refuses to build on the resulting go.sum drift.
Re-tidy the scratch copy
+ # go-sdk copied in above, which is the upstream-main fallback on a
checkout that has none of its
+ # own. Those two go-sdks differ, and Go refuses to build on the go.sum
drift. Re-tidy the scratch copy
# first so the build reconciles to whichever go-sdk it is actually
compiled against; the committed
# go.sum is untouched and stays guarded by the check-go-example-mod-tidy
prek hook.
if native:
@@ -2714,8 +2708,8 @@ def _lang_sdk_build_java_jar(
Both gradle invocations run against ``java_sdk_source``; only ``-p`` stays
pointed at the local
``java_example``, which is test-harness code that keeps tracking the
checked-out branch. The two
- therefore only agree when the source is the local ``java-sdk/`` -- on a
run targeting a non-main
- branch the example is compiled against upstream main's SDK, so it must
stay compatible with it.
+ therefore only agree when the source is the local ``java-sdk/`` -- on the
upstream-main fallback
+ the example is compiled against upstream's SDK, so it must stay compatible
with it.
"""
java_dir = staging / "java-artifacts"
java_dir.mkdir(parents=True, exist_ok=True)
@@ -3069,7 +3063,7 @@ def _setup_lang_sdk_test(
) -> None:
"""Provision the lang-SDK coordinator env on an already-deployed
KubernetesExecutor cluster.
- Fetches go-sdk/java-sdk from upstream main, then builds the Go/Java
artifacts, the Java worker
+ Resolves the go-sdk/java-sdk sources, then builds the Go/Java artifacts,
the Java worker
image and deploys localstack in parallel, then serially uploads the
artifacts, applies the
config + secret, and helm-upgrades Airflow with the lang-SDK values.
"""
diff --git a/dev/breeze/tests/test_kubernetes_lang_sdk_commands.py
b/dev/breeze/tests/test_kubernetes_lang_sdk_commands.py
index 0f12758e249..a94976a44e7 100644
--- a/dev/breeze/tests/test_kubernetes_lang_sdk_commands.py
+++ b/dev/breeze/tests/test_kubernetes_lang_sdk_commands.py
@@ -20,14 +20,12 @@ from unittest import mock
import pytest
-from airflow_breeze.branch_defaults import AIRFLOW_BRANCH
from airflow_breeze.commands import kubernetes_commands
from airflow_breeze.commands.kubernetes_commands import (
_lang_sdk_build_go_bundle,
_lang_sdk_build_java_jar,
_lang_sdk_fetch_upstream_sdk_sources,
_lang_sdk_resolve_sdk_sources,
- _lang_sdk_target_branch,
_lang_sdk_upload_artifacts,
)
from airflow_breeze.utils import shared_options
@@ -379,51 +377,56 @@ class TestSetupLangSdkTestNativeSelection:
}
-class TestLangSdkTargetBranch:
+class TestLangSdkResolveSdkSources:
+ @pytest.fixture
+ def repo_root(self, tmp_path, monkeypatch):
+ root = tmp_path / "repo"
+ root.mkdir()
+ monkeypatch.setattr(kubernetes_commands, "AIRFLOW_ROOT_PATH", root)
+ return root
+
@pytest.mark.parametrize(
- ("env", "expected"),
+ "env",
[
- pytest.param({"GITHUB_BASE_REF": "main"}, "main",
id="pr-targeting-main"),
- pytest.param({"GITHUB_BASE_REF": "v3-3-test"}, "v3-3-test",
id="pr-targeting-release"),
- pytest.param({"DEFAULT_BRANCH": "v3-3-test"}, "v3-3-test",
id="ci-default-branch"),
+ pytest.param({}, id="no-branch-env"),
+ # A release-branch run must build the branch's own SDKs: upstream
main's speak a later
+ # supervisor_schema_version than that branch's task-SDK supervisor
knows.
pytest.param(
- {"GITHUB_BASE_REF": "main", "DEFAULT_BRANCH": "v3-3-test"},
- "main",
- id="pr-target-wins-over-default-branch",
+ {"GITHUB_BASE_REF": "v3-3-test", "DEFAULT_BRANCH":
"v3-3-test"},
+ id="release-branch-env",
),
- pytest.param({}, AIRFLOW_BRANCH,
id="falls-back-to-this-checkouts-branch"),
],
)
- def test_resolves_target_branch(self, env, expected, monkeypatch):
+ @mock.patch.object(kubernetes_commands,
"_lang_sdk_fetch_upstream_sdk_sources")
+ def test_checkout_with_both_sdks_builds_from_them(
+ self, mock_fetch, env, repo_root, tmp_path, monkeypatch
+ ):
monkeypatch.delenv("GITHUB_BASE_REF", raising=False)
monkeypatch.delenv("DEFAULT_BRANCH", raising=False)
for key, value in env.items():
monkeypatch.setenv(key, value)
-
- assert _lang_sdk_target_branch() == expected
-
- def test_empty_env_var_does_not_mask_the_fallback(self, monkeypatch):
- """GitHub sets GITHUB_BASE_REF to an empty string on non-PR events
(push, schedule)."""
- monkeypatch.setenv("GITHUB_BASE_REF", "")
- monkeypatch.setenv("DEFAULT_BRANCH", "v3-3-test")
-
- assert _lang_sdk_target_branch() == "v3-3-test"
-
-
-class TestLangSdkResolveSdkSources:
- @mock.patch.object(kubernetes_commands,
"_lang_sdk_fetch_upstream_sdk_sources")
- def test_targeting_main_uses_the_checked_out_branch(self, mock_fetch,
tmp_path, monkeypatch):
- monkeypatch.setenv("GITHUB_BASE_REF", "main")
+ (repo_root / "go-sdk").mkdir()
+ (repo_root / "java-sdk").mkdir()
go_sdk, java_sdk = _lang_sdk_resolve_sdk_sources(tmp_path, None)
- assert go_sdk == kubernetes_commands.AIRFLOW_ROOT_PATH / "go-sdk"
- assert java_sdk == kubernetes_commands.AIRFLOW_ROOT_PATH / "java-sdk"
+ assert (go_sdk, java_sdk) == (repo_root / "go-sdk", repo_root /
"java-sdk")
mock_fetch.assert_not_called()
+ @pytest.mark.parametrize(
+ "present",
+ [
+ pytest.param((), id="neither-sdk"),
+ pytest.param(("go-sdk",), id="only-go-sdk"),
+ pytest.param(("java-sdk",), id="only-java-sdk"),
+ ],
+ )
@mock.patch.object(kubernetes_commands,
"_lang_sdk_fetch_upstream_sdk_sources")
- def test_targeting_another_branch_falls_back_to_upstream_main(self,
mock_fetch, tmp_path, monkeypatch):
- monkeypatch.setenv("GITHUB_BASE_REF", "v3-3-test")
+ def test_checkout_without_both_sdks_falls_back_to_upstream_main(
+ self, mock_fetch, present, repo_root, tmp_path
+ ):
+ for name in present:
+ (repo_root / name).mkdir()
mock_fetch.return_value = (tmp_path / "go-sdk", tmp_path / "java-sdk")
go_sdk, java_sdk = _lang_sdk_resolve_sdk_sources(tmp_path, None)
diff --git a/kubernetes-tests/lang_sdk/README.md
b/kubernetes-tests/lang_sdk/README.md
index 570ec2b67f0..93583ecbd8b 100644
--- a/kubernetes-tests/lang_sdk/README.md
+++ b/kubernetes-tests/lang_sdk/README.md
@@ -65,31 +65,32 @@ The Go binary, Java jar, and stub Dag share one object
store (localstack) but li
## Which SDK sources get built
-`go-sdk/` and `java-sdk/` are only developed on `main`; a release/backport
branch may lack them
-entirely or carry a stale, branch-cut-frozen copy. So `breeze k8s
setup-lang-sdk-test` (and
-`run-complete-tests --lang-sdk-test`) picks the sources from the branch the
run **targets**, resolved
-by `_lang_sdk_resolve_sdk_sources()` in `kubernetes_commands.py`:
+`breeze k8s setup-lang-sdk-test` (and `run-complete-tests --lang-sdk-test`)
resolves them in
+`_lang_sdk_resolve_sdk_sources()` in `kubernetes_commands.py`:
-| Target branch | Go/Java SDK sources |
+| Checkout | Go/Java SDK sources |
| --- | --- |
-| `main` | the checked-out branch's own `go-sdk/` and `java-sdk/` |
-| anything else (`v3-*-test`, …) | upstream `main`, fetched fresh via
`_lang_sdk_fetch_upstream_sdk_sources()` |
-
-The target is `GITHUB_BASE_REF` for a PR, then `DEFAULT_BRANCH` if set,
falling back to this
-checkout's own `AIRFLOW_BRANCH`. Building a main-targeting PR's own SDK is
what makes the k8s test
-exercise that PR: `go_example`/`java_example` are harness fixtures that track
the checked-out branch,
-so compiling them against a *different* SDK means any SDK rename in the PR
fails to build. A
-backport to a release-test branch still gets current SDK code, as before.
+| has `go-sdk/` and `java-sdk/` | its own copies |
+| has neither (or only one) | upstream `main`, fetched fresh via
`_lang_sdk_fetch_upstream_sdk_sources()` |
+
+The checkout's own copies win because they are the ones that pair with the
Airflow this test
+deploys. A packed bundle declares a dated `supervisor_schema_version` and the
task-SDK supervisor
+rejects a bundle whose version it does not know, so a release branch's Airflow
cannot run an SDK
+built from a later `main` — the Go task fails with `cannot find executable
bundle with usable
+supervisor_schema_version`. Building the checkout's own SDK is also what makes
the k8s test exercise
+a PR's SDK changes: `go_example`/`java_example` are harness fixtures that
track the checked-out
+branch, so compiling them against a *different* SDK means any SDK rename in
the PR fails to build.
+
+The upstream-`main` fallback is only for a branch cut before
`go-sdk`/`java-sdk` existed. When it
+kicks in, that copy and the branch's `go_example` can diverge (upstream may
change go-sdk's
+dependency graph while `go_example`'s committed `go.sum` is tidied against the
in-repo go-sdk), so
+the Go bundle build re-runs `go mod tidy` in its scratch workspace before
packing and reconciles to
+whichever `go-sdk` it is compiled against. The committed `go_example` `go.sum`
is untouched and
+stays guarded by the `check-go-example-mod-tidy` prek hook.
Everything else — `airflow-core/`, `task-sdk/`, the deployed Airflow image,
and this directory's own
`go_example`/`java_example` fixtures — always comes from the checked-out
branch.
-When the sources do come from upstream main, that copy and the branch's
`go_example` can diverge (a
-branch may change go-sdk's dependency graph while `go_example`'s committed
`go.sum` is tidied against
-the in-repo go-sdk), so the Go bundle build re-runs `go mod tidy` in its
scratch workspace before
-packing and reconciles to whichever `go-sdk` it is compiled against. The
committed `go_example`
-`go.sum` is untouched and stays guarded by the `check-go-example-mod-tidy`
prek hook.
-
## Running it
The artifacts, localstack, config, and Helm release are provisioned by a
single breeze
