This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 2e2132c3af5 Build lang-SDK k8s test artifacts from the branch when
targeting main (#71527)
2e2132c3af5 is described below
commit 2e2132c3af5baecb201a13add600d20b584ef9f6
Author: Jason(Zhe-You) Liu <[email protected]>
AuthorDate: Fri Aug 14 12:31:32 2026 +0800
Build lang-SDK k8s test artifacts from the branch when targeting main
(#71527)
* Build lang-SDK k8s test artifacts from the branch when targeting main
The k8s lang-SDK test compiled go_example/java_example -- harness fixtures
that
track the checked-out branch -- against upstream main's SDK
unconditionally, so
a PR that changes the Go/Java SDK was never exercised by the test that is
meant
to cover it, and any SDK rename failed the build outright.
Building from upstream main exists for release/backport branches, which may
lack
go-sdk/java-sdk or carry a branch-cut-frozen copy. That reason only applies
when
the run targets such a branch, so scope it to that case.
* Address lang-SDK k8s target-branch review feedback
Separates the upstream-fetch ref from the local-sources sentinel so
the two independent meanings don't share one constant, fixes the
"falls back to this checkout's branch" test case to assert against
AIRFLOW_BRANCH instead of a hardcoded "main" (which fails on backport
branches like v3-3-test), and stops claiming DEFAULT_BRANCH is set by
CI for this host-side lookup since nothing currently wires it in.
---
.../airflow_breeze/commands/kubernetes_commands.py | 71 ++++++++++++++++------
.../tests/test_kubernetes_lang_sdk_commands.py | 70 ++++++++++++++++++---
kubernetes-tests/lang_sdk/README.md | 38 +++++++-----
3 files changed, 139 insertions(+), 40 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
b/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
index c064e7400b3..47d1cf3710a 100644
--- a/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
@@ -32,6 +32,7 @@ from typing import Any
import click
import yaml
+from airflow_breeze.branch_defaults import AIRFLOW_BRANCH
from airflow_breeze.commands.common_options import (
option_answer,
option_debug_resources,
@@ -2509,10 +2510,41 @@ LANG_SDK_AWS_CONN_URI = (
"aws://test:test@/?region_name=us-east-1&"
"endpoint_url=http%3A%2F%2Flocalstack.airflow.svc.cluster.local%3A4566"
)
-# The Go/Java SDKs are always built from upstream main so branches with stale
or missing
-# go-sdk/java-sdk copies still test current SDK sources. See
kubernetes-tests/lang_sdk/README.md.
+# Runs targeting a branch other than main build the Go/Java SDKs from upstream
main, so
+# release/backport branches with stale or missing go-sdk/java-sdk copies still
test current SDK
+# sources. See kubernetes-tests/lang_sdk/README.md.
LANG_SDK_UPSTREAM_GIT_URL = "https://github.com/apache/airflow.git";
LANG_SDK_UPSTREAM_REF = "main"
+# The target branch for which the lang-SDK artifacts are built from this
checkout's own SDK
+# sources rather than fetched from upstream. Distinct from
LANG_SDK_UPSTREAM_REF (the ref fetched
+# from upstream) even though both are "main" today -- one names a git ref, the
other a sentinel.
+LANG_SDK_LOCAL_SOURCE_BRANCH = "main"
+
+
+def _lang_sdk_target_branch() -> str:
+ """Branch this run targets: GITHUB_BASE_REF or DEFAULT_BRANCH if set, else
this checkout's."""
+ return os.environ.get("GITHUB_BASE_REF") or
os.environ.get("DEFAULT_BRANCH") or AIRFLOW_BRANCH
+
+
+def _lang_sdk_resolve_sdk_sources(staging: Path, output: Output | None) ->
tuple[Path, Path]:
+ """Resolve the go-sdk/java-sdk trees the lang-SDK artifacts are built from.
+
+ A run targeting main builds the checked-out branch's own SDK sources, so a
PR's SDK changes
+ are what the k8s test exercises -- without this,
``java_example``/``go_example`` (harness code
+ that tracks the branch) is compiled against a different SDK than the
branch it belongs to, and
+ any SDK rename breaks the build. Runs targeting anything else fall back to
upstream main.
+ """
+ target = _lang_sdk_target_branch()
+ if target == LANG_SDK_LOCAL_SOURCE_BRANCH:
+ get_console(output=output).print(
+ f"[info]Run targets {target}: building the lang-SDK Go/Java
artifacts from this branch"
+ )
+ return AIRFLOW_ROOT_PATH / "go-sdk", AIRFLOW_ROOT_PATH / "java-sdk"
+ get_console(output=output).print(
+ f"[info]Run targets {target}, not {LANG_SDK_LOCAL_SOURCE_BRANCH}:
building the lang-SDK Go/Java "
+ f"artifacts from upstream {LANG_SDK_UPSTREAM_REF}"
+ )
+ return _lang_sdk_fetch_upstream_sdk_sources(staging, output)
def _lang_sdk_fetch_upstream_sdk_sources(staging: Path, output: Output | None)
-> tuple[Path, Path]:
@@ -2582,7 +2614,7 @@ def _lang_sdk_fetch_upstream_sdk_sources(staging: Path,
output: Output | None) -
def _lang_sdk_build_go_bundle(
- staging: Path, upstream_go_sdk: Path, output: Output | None, *, native:
bool = False
+ staging: Path, go_sdk_source: Path, output: Output | None, *, native: bool
= False
) -> None:
"""Build the Go bundle into ``staging/go-artifacts`` and copy the result
into the staging dir.
@@ -2592,10 +2624,10 @@ def _lang_sdk_build_go_bundle(
skipping the container image pull and reusing the runner's module/build
cache.
go_example's go.mod ``replace``s go-sdk by relative path, so the build
runs in a scratch
- workspace mirroring the repo layout with ``upstream_go_sdk`` at
``<workspace>/go-sdk``,
- letting the unmodified directive resolve against the upstream copy. The
scratch go_example is
- re-tidied before packing so its go.sum reconciles to that upstream go-sdk
(which may differ from
- the in-repo go-sdk its committed go.sum was tidied against).
+ workspace mirroring the repo layout with ``go_sdk_source`` at
``<workspace>/go-sdk``, letting
+ the unmodified directive resolve against it. The scratch go_example is
re-tidied before packing
+ so its go.sum reconciles to that go-sdk, which differs from the in-repo
one its committed go.sum
+ was tidied against whenever the source is upstream main.
"""
go_dir = staging / "go-artifacts"
go_dir.mkdir(parents=True, exist_ok=True)
@@ -2604,10 +2636,10 @@ def _lang_sdk_build_go_bundle(
example_path = workspace / example_rel
example_path.parent.mkdir(parents=True, exist_ok=True)
shutil.copytree(LANG_SDK_GO_EXAMPLE_PATH, example_path,
ignore=shutil.ignore_patterns(".home"))
- # In dry-run the fetch/extract commands are skipped, so the upstream copy
and build outputs
+ # In dry-run the fetch/extract commands are skipped, so the extracted copy
and build outputs
# never materialize -- skip the filesystem work that depends on them.
if not get_dry_run():
- shutil.copytree(upstream_go_sdk, workspace / "go-sdk")
+ shutil.copytree(go_sdk_source, workspace / "go-sdk")
output_bin = example_path / "bin" / LANG_SDK_GO_BUNDLE_NAME
output_bin.parent.mkdir(parents=True, exist_ok=True)
@@ -2677,7 +2709,7 @@ def _lang_sdk_build_go_bundle(
def _lang_sdk_build_java_jar(
- staging: Path, upstream_java_sdk: Path, output: Output | None, *, native:
bool = False
+ staging: Path, java_sdk_source: Path, output: Output | None, *, native:
bool = False
) -> None:
"""Publish the Java SDK to mavenLocal then build the java_example jar into
``staging/java-artifacts``.
@@ -2688,9 +2720,10 @@ def _lang_sdk_build_java_jar(
cache. ``java_example`` resolves the SDK from ``mavenLocal()``, so the SDK
is published first, then
the bundle is built with java-sdk's gradle wrapper pointed at the example
project (``-p``).
- Both gradle invocations run against ``upstream_java_sdk`` rather than the
local ``java-sdk/``;
- only ``-p`` stays pointed at the local ``java_example``, which is
test-harness code that keeps
- tracking the checked-out branch.
+ Both gradle invocations run against ``java_sdk_source``; only ``-p`` stays
pointed at the local
+ ``java_example``, which is test-harness code that keeps tracking the
checked-out branch. The two
+ therefore only agree when the source is the local ``java-sdk/`` -- on a
run targeting a non-main
+ branch the example is compiled against upstream main's SDK, so it must
stay compatible with it.
"""
java_dir = staging / "java-artifacts"
java_dir.mkdir(parents=True, exist_ok=True)
@@ -2701,14 +2734,14 @@ def _lang_sdk_build_java_jar(
)
run_command(
["./gradlew", "publishToMavenLocal", "-PskipSigning=true",
"--no-daemon", "--console=plain"],
- cwd=upstream_java_sdk,
+ cwd=java_sdk_source,
output=output,
check=True,
)
get_console(output=output).print("[info]Building Java jar with the
host Gradle toolchain")
run_command(
["./gradlew", "-p", str(LANG_SDK_JAVA_EXAMPLE_PATH), "bundle",
"--no-daemon", "--console=plain"],
- cwd=upstream_java_sdk,
+ cwd=java_sdk_source,
output=output,
check=True,
)
@@ -2738,7 +2771,7 @@ def _lang_sdk_build_java_jar(
"-v",
f"{AIRFLOW_ROOT_PATH}:/repo",
"-v",
- f"{upstream_java_sdk}:/repo/java-sdk",
+ f"{java_sdk_source}:/repo/java-sdk",
]
get_console(output=output).print("[info]Publishing Java SDK artifacts
to local Maven repository")
run_command(
@@ -3059,15 +3092,15 @@ def _setup_lang_sdk_test(
native = os.environ.get("LANG_SDK_NATIVE_TOOLCHAIN", "").lower() == "true"
with tempfile.TemporaryDirectory(prefix="lang_sdk_artifacts_") as tmp:
staging = Path(tmp)
- upstream_go_sdk, upstream_java_sdk =
_lang_sdk_fetch_upstream_sdk_sources(staging, output)
+ go_sdk_source, java_sdk_source =
_lang_sdk_resolve_sdk_sources(staging, output)
steps: list[tuple[str, Callable[[Output | None], Any]]] = [
(
"Build Go bundle",
- lambda o: _lang_sdk_build_go_bundle(staging, upstream_go_sdk,
o, native=native),
+ lambda o: _lang_sdk_build_go_bundle(staging, go_sdk_source, o,
native=native),
),
(
"Build Java jar",
- lambda o: _lang_sdk_build_java_jar(staging, upstream_java_sdk,
o, native=native),
+ lambda o: _lang_sdk_build_java_jar(staging, java_sdk_source,
o, native=native),
),
("Deploy localstack", lambda o:
_lang_sdk_deploy_localstack(python, kubernetes_version, o)),
]
diff --git a/dev/breeze/tests/test_kubernetes_lang_sdk_commands.py
b/dev/breeze/tests/test_kubernetes_lang_sdk_commands.py
index 31332c24d16..0f12758e249 100644
--- a/dev/breeze/tests/test_kubernetes_lang_sdk_commands.py
+++ b/dev/breeze/tests/test_kubernetes_lang_sdk_commands.py
@@ -20,11 +20,14 @@ from unittest import mock
import pytest
+from airflow_breeze.branch_defaults import AIRFLOW_BRANCH
from airflow_breeze.commands import kubernetes_commands
from airflow_breeze.commands.kubernetes_commands import (
_lang_sdk_build_go_bundle,
_lang_sdk_build_java_jar,
_lang_sdk_fetch_upstream_sdk_sources,
+ _lang_sdk_resolve_sdk_sources,
+ _lang_sdk_target_branch,
_lang_sdk_upload_artifacts,
)
from airflow_breeze.utils import shared_options
@@ -325,8 +328,8 @@ class TestSetupLangSdkTestNativeSelection:
monkeypatch.setenv("LANG_SDK_NATIVE_TOOLCHAIN", env_value)
captured: dict[str, bool] = {}
- fake_go_sdk = tmp_path / "upstream_go_sdk"
- fake_java_sdk = tmp_path / "upstream_java_sdk"
+ fake_go_sdk = tmp_path / "resolved_go_sdk"
+ fake_java_sdk = tmp_path / "resolved_java_sdk"
def fake_parallel(steps, output):
for _title, thunk in steps:
@@ -335,21 +338,21 @@ class TestSetupLangSdkTestNativeSelection:
monkeypatch.setattr(kubernetes_commands, "_run_lang_sdk_parallel",
fake_parallel)
monkeypatch.setattr(
kubernetes_commands,
- "_lang_sdk_fetch_upstream_sdk_sources",
+ "_lang_sdk_resolve_sdk_sources",
lambda staging, output: (fake_go_sdk, fake_java_sdk),
)
monkeypatch.setattr(
kubernetes_commands,
"_lang_sdk_build_go_bundle",
- lambda staging, upstream_go_sdk, output, *, native:
captured.update(
- go=native, go_sdk=upstream_go_sdk
+ lambda staging, go_sdk_source, output, *, native: captured.update(
+ go=native, go_sdk=go_sdk_source
),
)
monkeypatch.setattr(
kubernetes_commands,
"_lang_sdk_build_java_jar",
- lambda staging, upstream_java_sdk, output, *, native:
captured.update(
- java=native, java_sdk=upstream_java_sdk
+ lambda staging, java_sdk_source, output, *, native:
captured.update(
+ java=native, java_sdk=java_sdk_source
),
)
for name in (
@@ -374,3 +377,56 @@ class TestSetupLangSdkTestNativeSelection:
"go_sdk": fake_go_sdk,
"java_sdk": fake_java_sdk,
}
+
+
+class TestLangSdkTargetBranch:
+ @pytest.mark.parametrize(
+ ("env", "expected"),
+ [
+ pytest.param({"GITHUB_BASE_REF": "main"}, "main",
id="pr-targeting-main"),
+ pytest.param({"GITHUB_BASE_REF": "v3-3-test"}, "v3-3-test",
id="pr-targeting-release"),
+ pytest.param({"DEFAULT_BRANCH": "v3-3-test"}, "v3-3-test",
id="ci-default-branch"),
+ pytest.param(
+ {"GITHUB_BASE_REF": "main", "DEFAULT_BRANCH": "v3-3-test"},
+ "main",
+ id="pr-target-wins-over-default-branch",
+ ),
+ pytest.param({}, AIRFLOW_BRANCH,
id="falls-back-to-this-checkouts-branch"),
+ ],
+ )
+ def test_resolves_target_branch(self, env, expected, monkeypatch):
+ monkeypatch.delenv("GITHUB_BASE_REF", raising=False)
+ monkeypatch.delenv("DEFAULT_BRANCH", raising=False)
+ for key, value in env.items():
+ monkeypatch.setenv(key, value)
+
+ assert _lang_sdk_target_branch() == expected
+
+ def test_empty_env_var_does_not_mask_the_fallback(self, monkeypatch):
+ """GitHub sets GITHUB_BASE_REF to an empty string on non-PR events
(push, schedule)."""
+ monkeypatch.setenv("GITHUB_BASE_REF", "")
+ monkeypatch.setenv("DEFAULT_BRANCH", "v3-3-test")
+
+ assert _lang_sdk_target_branch() == "v3-3-test"
+
+
+class TestLangSdkResolveSdkSources:
+ @mock.patch.object(kubernetes_commands,
"_lang_sdk_fetch_upstream_sdk_sources")
+ def test_targeting_main_uses_the_checked_out_branch(self, mock_fetch,
tmp_path, monkeypatch):
+ monkeypatch.setenv("GITHUB_BASE_REF", "main")
+
+ go_sdk, java_sdk = _lang_sdk_resolve_sdk_sources(tmp_path, None)
+
+ assert go_sdk == kubernetes_commands.AIRFLOW_ROOT_PATH / "go-sdk"
+ assert java_sdk == kubernetes_commands.AIRFLOW_ROOT_PATH / "java-sdk"
+ mock_fetch.assert_not_called()
+
+ @mock.patch.object(kubernetes_commands,
"_lang_sdk_fetch_upstream_sdk_sources")
+ def test_targeting_another_branch_falls_back_to_upstream_main(self,
mock_fetch, tmp_path, monkeypatch):
+ monkeypatch.setenv("GITHUB_BASE_REF", "v3-3-test")
+ mock_fetch.return_value = (tmp_path / "go-sdk", tmp_path / "java-sdk")
+
+ go_sdk, java_sdk = _lang_sdk_resolve_sdk_sources(tmp_path, None)
+
+ mock_fetch.assert_called_once_with(tmp_path, None)
+ assert (go_sdk, java_sdk) == (tmp_path / "go-sdk", tmp_path /
"java-sdk")
diff --git a/kubernetes-tests/lang_sdk/README.md
b/kubernetes-tests/lang_sdk/README.md
index 52e081c197f..570ec2b67f0 100644
--- a/kubernetes-tests/lang_sdk/README.md
+++ b/kubernetes-tests/lang_sdk/README.md
@@ -63,22 +63,32 @@ coordinator scans.
The Go binary, Java jar, and stub Dag share one object store (localstack) but
live in
**separate buckets** (`go-artifacts`, `java-artifacts`, `dags`).
-## SDK sources always come from upstream main
+## Which SDK sources get built
+
+`go-sdk/` and `java-sdk/` are only developed on `main`; a release/backport
branch may lack them
+entirely or carry a stale, branch-cut-frozen copy. So `breeze k8s
setup-lang-sdk-test` (and
+`run-complete-tests --lang-sdk-test`) picks the sources from the branch the
run **targets**, resolved
+by `_lang_sdk_resolve_sdk_sources()` in `kubernetes_commands.py`:
+
+| Target branch | Go/Java SDK sources |
+| --- | --- |
+| `main` | the checked-out branch's own `go-sdk/` and `java-sdk/` |
+| anything else (`v3-*-test`, …) | upstream `main`, fetched fresh via
`_lang_sdk_fetch_upstream_sdk_sources()` |
+
+The target is `GITHUB_BASE_REF` for a PR, then `DEFAULT_BRANCH` if set,
falling back to this
+checkout's own `AIRFLOW_BRANCH`. Building a main-targeting PR's own SDK is
what makes the k8s test
+exercise that PR: `go_example`/`java_example` are harness fixtures that track
the checked-out branch,
+so compiling them against a *different* SDK means any SDK rename in the PR
fails to build. A
+backport to a release-test branch still gets current SDK code, as before.
-`go-sdk/` and `java-sdk/` are young, fast-moving directories that a
release/backport branch may lack
-entirely or carry a stale, branch-cut-frozen copy of. So regardless of which
branch/ref this repo is
-checked out at, `breeze k8s setup-lang-sdk-test` (and `run-complete-tests
--lang-sdk-test`) always
-builds the Go bundle and Java jar from upstream `main`'s `go-sdk`/`java-sdk` —
fetched fresh via
-`_lang_sdk_fetch_upstream_sdk_sources()` in `kubernetes_commands.py`, never
from whatever's on disk.
Everything else — `airflow-core/`, `task-sdk/`, the deployed Airflow image,
and this directory's own
-`go_example`/`java_example` harness fixtures — still comes from the
checked-out branch as before, so a
-backport of a core/task-sdk fix to a release-test branch keeps testing against
current SDK code.
-
-Because the branch's `go_example` and upstream main's `go-sdk` can diverge (a
branch may change
-go-sdk's dependency graph while `go_example`'s committed `go.sum` is tidied
against the in-repo
-go-sdk), the Go bundle build re-runs `go mod tidy` in its scratch workspace
before packing, so the
-build reconciles to whichever `go-sdk` it is compiled against. The committed
`go_example` `go.sum`
-is untouched and stays guarded by the `check-go-example-mod-tidy` prek hook.
+`go_example`/`java_example` fixtures — always comes from the checked-out
branch.
+
+When the sources do come from upstream main, that copy and the branch's
`go_example` can diverge (a
+branch may change go-sdk's dependency graph while `go_example`'s committed
`go.sum` is tidied against
+the in-repo go-sdk), so the Go bundle build re-runs `go mod tidy` in its
scratch workspace before
+packing and reconciles to whichever `go-sdk` it is compiled against. The
committed `go_example`
+`go.sum` is untouched and stays guarded by the `check-go-example-mod-tidy`
prek hook.
## Running it
