kaxil commented on code in PR #67153: URL: https://github.com/apache/airflow/pull/67153#discussion_r3295554985
########## go-sdk/adr/0002-use-go-tool-directive-for-bundle-packer.md: ########## @@ -0,0 +1,222 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 2. Use the Go 1.24 `tool` directive to deliver the bundle packer + +Date: 2026-04-30 + +## Status + +Accepted. Selects from the option register in +[ADR 0001](0001-bundle-packing-options.md). + +The output-format portion of this ADR (the packer writes a ZIP archive +following the bundle spec) is superseded by +[ADR 0004](0004-self-contained-executable-bundle.md): the packer now +writes a self-contained executable with an appended footer carrying +the source bytes and the manifest. The packer's *mechanism* (Option +A standalone binary + Option D introspection contract + Option H +`tool` directive) is unchanged. The decision sketches below mention +ZIP output; read them with the ADR 0004 substitution in mind, and +treat ADR 0004 as authoritative wherever the two disagree. + +## Context + +[ADR 0001](0001-bundle-packing-options.md) enumerated nine candidate +mechanisms for producing a conforming bundle ZIP +([`task-sdk/docs/bundle-spec.rst`](../../task-sdk/docs/bundle-spec.rst)) +from a Go SDK build. Two reasons drive the choice: + +1. **The repository already requires Go 1.24.** `go-sdk/go.mod` declares + `go 1.24.0` with `toolchain go1.24.6`, so language features added in + 1.24 are available to every consumer of the SDK by construction. +2. **Contributors expect Go-native workflows.** The Go 1.24 `tool` + directive is the toolchain's native answer to "depend on a + build-time CLI without polluting the global PATH." It pins the tool + version per-project in `go.mod`, resolves it through the standard + module cache, and exposes it as `go tool <name>`, with no extra + installer and no per-worktree drift. The same problem on the Python + side led `breeze` to switch to `uvx` in + [ADR 0017](../../dev/breeze/doc/adr/0017-use-uvx-to-run-breeze-from-local-sources.md); + `tool` is the analogous answer here. + +The `tool` directive is a delivery mechanism. It still needs an +underlying implementation. We pair it with two of the implementation +options from ADR 0001, with a UX twist: + +- **Option A (standalone packer):** a single-purpose binary at + `go-sdk/cmd/airflow-go-pack`. It still operates as one process with + a clear input/output contract, but it drives `go build` internally + by default so that the common case is one command: + `go tool airflow-go-pack ./pkg`. Authors who already produce their + own binary can opt out via `--executable <path>` and skip the build + phase. This is closer to Option B's ergonomics than the original + ADR 0001 sketch, but kept inside the standalone-packer shape so the + SDK does not own a fully general `go build` wrapper. Review Comment: *"kept inside the standalone-packer shape so the SDK does not own a fully general `go build` wrapper"* reads like having it both ways. The packer drives `go build` internally with arbitrary `--` flag passthrough; in practice the SDK does own a `go build` wrapper -- the "fully general" qualifier is carrying the distinction. What's the line being drawn? If it's *"no first-class subcommands beyond `pack`"*, say that directly; the current framing reads like a hedge against ADR 0001's rejection of Option B. The next reader (or AIP-108 reviewer) is going to ask the same question. ########## go-sdk/adr/0004-self-contained-executable-bundle.md: ########## @@ -0,0 +1,304 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 4. Self-contained executable bundle (footer-embedded source and metadata) + +Date: 2026-05-04 + +## Status + +Accepted. Supersedes the ZIP-archive container portion of +[ADR 0001](0001-bundle-packing-options.md) and the ZIP output sketched +in [ADR 0002](0002-use-go-tool-directive-for-bundle-packer.md). The +packer mechanism (Option A standalone packer + Option D introspection +contract + Option H `tool` directive) is unchanged; only the artefact +the packer writes is changed. + +## Context + +ADR 0001 / ADR 0002 picked a ZIP archive as the bundle container, +following the executable provider's existing +[`task-sdk/docs/bundle-spec.rst`](../../task-sdk/docs/bundle-spec.rst). +A conforming bundle today is `bundle.zip` with three required entries: +`airflow-metadata.yaml`, the primary DAG source file, and the compiled +executable. + +That layout has three properties we want to preserve: + +1. **Discovery without execution.** The scanner must be able to read + `dag_id` / `task_id` and the SDK language/version from a bundle on + disk without running the binary. ADR 0002 already enforces this — + `airflow-go-pack` runs the binary once at build time, captures its + `--dump-bundle-spec` output into the manifest, and the scanner reads + the manifest at deploy time. +2. **Source available for the UI.** The Airflow UI's source-view + panel needs to render the DAG file. The current spec ships it as a + verbatim ZIP entry referenced by the manifest's `source` field. +3. **Single deployment unit.** Drop one file in + `[executable] bundles_folder` and the scanner picks it up. + +What the ZIP container costs us: + +- **Two artefacts in flight.** `go build` produces a binary; the + packer wraps it into a ZIP. Anything that touches the binary after + it is wrapped (re-strip, re-sign, swap-in a debug build) drifts from + the manifest unless the wrapping is redone. The wrapping step is + cheap but the drift mode is real. +- **A second container format on the consumer side.** The scanner + must open archives, find members by name, and materialise the + executable into a transient cache before the runtime can exec it. + That is `archive/zip` on the Python side plus a per-bundle cache + directory. +- **Inspection requires a different tool than running.** `unzip` to + inspect, then run; or run, then `unzip` to debug. Two muscle memories. + +Native-executable SDKs (Go, Rust, C++, Zig) all produce a single +self-contained binary by static linking. The binary itself is already +the only thing that has to land on the worker host to run a task. The +manifest and the source file are small data the scanner needs but the +runtime doesn't. Both can ride along in a footer appended to the +binary, with the binary remaining a runnable executable. + +This is the same pattern self-extracting installers, `goreleaser`-style +self-update images, and embedded-asset binaries already use: append +data after the OS-recognised binary structure, leave a fixed-size +trailer at the very end so a reader can locate the data, and validate +with a magic value. + +The user-facing claim becomes "the executable *is* the bundle." A +bundle directory looks like: + +``` +/opt/airflow/executable-bundles/ +├── example +├── pipeline +└── analytics +``` + +(Filenames follow OS conventions: no extension on Linux/macOS, `.exe` +on Windows. The scanner identifies bundles by the trailer's magic, not +by the filename.) + +## Decision + +Replace the bundle's ZIP container with a footer appended to the +compiled executable. The executable's normal byte content is unchanged +and it remains directly runnable; the footer is data that follows the +last byte the OS loader cares about. + +### Footer layout + +A bundle file is laid out as: + +```text ++---------------------------------+ +| <native executable: ELF/Mach-O/PE, | +| including any code-signing structures> | ++---------------------------------+ <- end of "binary" region +| source bytes (variable length) | raw root source file, UTF-8, +| | length = source_len; MAY be 0 ++---------------------------------+ +| metadata bytes (variable length)| airflow-metadata.yaml content, +| | UTF-8, length = metadata_len ++---------------------------------+ +| trailer (32 bytes, little-endian fixed layout): | +| bytes 0..3 source_len u32 | +| bytes 4..7 metadata_len u32 | +| bytes 8..11 footer_ver u32 (= 1) | +| bytes 12..23 reserved 12 bytes, zero | +| bytes 24..31 magic 8 bytes ASCII "AFBNDL01" | ++---------------------------------+ <- EOF +``` + +`AFBNDL01` is `0x41 0x46 0x42 0x4E 0x44 0x4C 0x30 0x31`. The two +trailing ASCII digits are the footer-format version, repeated for human +inspection (`tail -c 8 ./mybundle | xxd`); the binary `footer_ver` +field is the source of truth for parsing. + +Reader algorithm: + +1. Open the file. Seek to `EOF - 32`. Read 32 bytes. +2. Compare bytes 24..31 against `AFBNDL01`. If different, the file is + not a bundle; the scanner ignores it. +3. Parse `footer_ver`. If unknown, fail with a versioning error. +4. Compute `metadata_start = filesize - 32 - metadata_len` and + `source_start = metadata_start - source_len`. +5. Read `metadata_len` bytes from `metadata_start` for the manifest. +6. Read `source_len` bytes from `source_start` for the source view. + If `source_len == 0`, no source is embedded; the UI falls back to + "(source not available)". +7. Validate that `source_start >= 0` and that the implied "binary + region" (bytes `[0, source_start)`) is non-empty. + +Ordering note: source comes *before* metadata so a future +`format_version` can introduce extra trailing blobs (e.g. signed +checksums, compressed deps) by extending the trailer rather than +inserting between existing blobs. + +### Manifest schema changes + +The manifest content is the same YAML as today, with two field-level +changes that follow from the footer container: + +- **Drop `executable`.** The binary *is* the file; there is no + archive-relative path to record. +- **Redefine `source` as a display filename, not a path.** The source + bytes live in the footer; the manifest's `source` field carries the + original filename (e.g. `example.go`) so the UI can show it as a + filename in the source-view panel and pick a syntax-highlighting + mode from the extension. + +Everything else (`format_version`, `sdk.language`, `sdk.version`, +`dags`, the open-additivity rule for unknown keys) is unchanged. + +### Build pipeline + +The packer's behaviour from ADR 0002 changes only at the final write +step: + +1. Resolve target package, locate the file with `func main()`. (No + change.) +2. Run `go build [forwarded flags] -o <out> <pkg>`. (No change.) +3. Exec the freshly built binary with `--dump-bundle-spec` to obtain + the manifest. (No change.) +4. **New:** read the source file's bytes; serialise the manifest to + YAML; append `<source><metadata><trailer>` to `<out>`. +5. Default output path becomes `<bundleName>` (or `<bundleName>.exe` + on Windows), not `<bundleName>.zip`. + +Ordering against post-build steps: + +- **Strip:** must run *before* append. Stripping a file that already + has a footer either leaves the footer intact (most strip + implementations stop at the OS-defined end of the binary) or + truncates it; do not rely on either. +- **Code-sign:** must run *after* append on platforms whose signature + covers the entire file (Linux dm-verity, macOS post-Big-Sur for + certain notarisation flows, Windows Authenticode). The signature Review Comment: The code-sign claim is platform-conditional, not universal: - **Linux dm-verity / macOS codesign**: signature covers the full file bytes including any trailer, so `append footer -> sign binary` works. - **Windows Authenticode**: the signature is itself a `WIN_CERTIFICATE` structure at the *end* of the PE, referenced by the optional header's security directory. Appending a footer **after** signing puts the trailer outside the signed region (the OS still loads the binary, since the security directory offset/size is unchanged, but the trailer bytes are unsigned). Either call this out per-platform in the ADR, or document that Windows needs a different layout (footer inside a signed PE section, signed second) so the implementation doesn't have to discover this after shipping. Same applies if a future SDK targets iOS/Android (code-sign-with-entitlements has similar constraints). ########## go-sdk/adr/0003-coordinator-protocol-msgpack-ipc.md: ########## @@ -0,0 +1,381 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 3. Dual-mode bundle binary: msgpack-over-IPC coordinator protocol alongside the existing go-plugin/Edge-Worker path + +Date: 2026-04-30 + +## Status + +Accepted. + +## Context + +A Go SDK bundle binary today (the artefact built from +[`go-sdk/example/bundle/main.go`](../example/bundle/main.go) via +`bundlev1server.Serve`) speaks exactly one protocol: HashiCorp +[`go-plugin`](https://github.com/hashicorp/go-plugin) gRPC over a +stdio-negotiated socket, gated by the magic-cookie handshake declared in +[`pkg/bundles/shared/handshake.go`](../pkg/bundles/shared/handshake.go). +The Airflow Go *Edge Worker* +([`cmd/airflow-go-edge-worker`](../cmd/airflow-go-edge-worker/main.go), +[`edge/`](../edge)) is the consumer of that protocol — it execs the +bundle binary as a child process, completes the go-plugin handshake, +opens the `DagBundle` gRPC client, and drives `GetMetadata`/`Execute` +([`bundle/bundlev1/bundlev1server/impl/plugin.go`](../bundle/bundlev1/bundlev1server/impl/plugin.go)). +The bundle binary never listens on a public socket; the protocol is +local-process only. + +Meanwhile, the Python side of Airflow has standardised on a different +wire protocol for non-Python language runtimes — the *coordinator +protocol* — pioneered by the Java SDK and described in +[java-sdk ADR 0002](../../java-sdk/adr/0002-dag-parsing.md) +and +[java-sdk ADR 0003](../../java-sdk/adr/0003-workload-execution.md). +Its shape is: + +- The runtime is launched with `--comm=<host:port>` and + `--logs=<host:port>` CLI arguments. +- It connects out (TCP, loopback) to both addresses. +- Frames on the comm channel are length-prefixed msgpack: a 4-byte + big-endian length followed by the msgpack payload. Requests are + `[id, body]`; responses are `[id, body, error]`. +- Two workloads share one channel, distinguished by the first inbound + frame: `DagFileParseRequest` (one-shot, returns + `DagFileParsingResult` and exits) or `StartupDetails` (multi-round + task execution: the runtime sends `GetConnection` / `GetVariable` / + `GetXCom` / `SetXCom` and terminates with `SucceedTask` or + `TaskState`). +- The logs channel carries structured JSON log records emitted by the + runtime. + +The Python-side launcher is +[`ExecutableCoordinator`](../../task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py), +which already builds command lines of the form +`<binary> --comm=<addr> --logs=<addr>` for both `dag_parsing_runtime_cmd` +and `task_execution_runtime_cmd`. The bundle-spec contract +([`task-sdk/docs/bundle-spec.rst`](../../task-sdk/docs/bundle-spec.rst)) +ratifies that any compiled SDK shipping a ZIP bundle "MUST honour the +SDK coordinator protocol (`--comm=<addr>` / `--logs=<addr>` +socket-based IPC)". The Java SDK satisfies this contract; the Go SDK +currently does not. + +The two protocols target different deployment shapes: + +- **go-plugin / Edge Worker.** The Go-native worker is itself a long-running + process that loads bundles in-process and dispatches tasks to them + over gRPC. It is the only consumer that speaks go-plugin to a Go + bundle today, and it owns the full task-runtime stack on the worker + host (no Python in the data path). This is the path + [`go-sdk/example/bundle/main.go`](../example/bundle/main.go) was + written for and the path that + [`pkg/worker`](../pkg/worker) drives. +- **Coordinator / `ExecutableCoordinator`.** The Python task + runner forks a child that runs `<binary> --comm=… --logs=…`, + bridges its socket to the Airflow supervisor's fd 0, and proxies + Airflow service calls (`GetConnection`, `GetVariable`, ...) through + to the Execution API. This is how Airflow runs non-Python tasks + *without* a per-language worker — the same way Java runs today, and + the same way Rust/C++/Zig will run in the future. It is also the + only path the executable provider's bundle spec recognises. + +Today these two paths require two different binaries, even though the +DAG/task definitions, the registry, the worker plumbing, and the +serialisation surfaces overlap almost entirely. That is the gap this +ADR closes. + +The user-written `main()` is one line — +`bundlev1server.Serve(&myBundle{})` — and we want to keep it one line. +Whichever protocol the binary should speak must be decided inside +`Serve` based on how it was invoked, not by branching in user code. + +## Decision + +Make the SDK bundle binary **dual-mode**. A single +`bundlev1server.Serve(bundle, opts...)` call dispatches to one of two +protocol servers based on its CLI arguments and process environment. +User code does not change. + +### Invocation matrix + +`Serve` parses flags first, then chooses a mode in this order: + +| Trigger | Mode | Behaviour | +|--------------------------------------------------------|-----------------|-----------| +| `--bundle-metadata` | metadata-dump | Existing flag (ADR 0001 / `server.go:37`). Prints `BundleInfo` JSON and exits. | +| `--dump-bundle-spec` | spec-dump | Existing flag added by [ADR 0002](0002-use-go-tool-directive-for-bundle-packer.md). Prints the full bundle spec JSON (`sdk`, `dags`) used by `airflow-go-pack`. | +| `--comm=<host:port> --logs=<host:port>` | **coordinator** | New. Speaks the msgpack-over-IPC coordinator protocol. Both flags are required; partial use is a hard error. | +| `AIRFLOW_BUNDLE_MAGIC_COOKIE` env var present (default) | go-plugin | Existing behaviour. Hands off to `plugin.Serve` which performs the handshake and serves `DagBundle` gRPC to the Edge Worker. | +| Otherwise | error | Print usage to stderr and exit non-zero. Today this case implicitly errors via go-plugin's failed handshake; we make the diagnostic explicit so authors running the binary directly get a clear message. | Review Comment: The dispatch matrix has `AIRFLOW_BUNDLE_MAGIC_COOKIE present (default)` -> go-plugin (line 124) and `Otherwise -> error` (this line). These overlap: which fires when nothing matches? Today, running the bare binary errors via go-plugin's failed handshake. The new dispatcher needs to either explicitly check the env var to differentiate go-plugin from "no mode selected", or drop the env-var gating and treat go-plugin as the catch-all (then `Otherwise` never triggers and the row is dead code). Worth resolving in the ADR before implementation -- right now both readings are defensible and they pick different behaviour. ########## go-sdk/adr/0001-bundle-packing-options.md: ########## @@ -0,0 +1,292 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 1. Post-build bundle-packing options for the Go SDK + +Date: 2026-04-30 + +## Status + +Accepted as the option register. The packer-mechanism decision is +recorded in [ADR 0002](0002-use-go-tool-directive-for-bundle-packer.md): +Option H (Go 1.24 `tool` directive) for delivery, paired with Option A +(standalone `airflow-go-pack` binary) and Option D (standardised +`--dump-bundle-spec` introspection contract). + +The container-format assumption running through this ADR — that the +output is a ZIP archive — is superseded by +[ADR 0004](0004-self-contained-executable-bundle.md), which embeds the +source and manifest in a footer appended to the executable. The +options below still describe valid *packer mechanisms*; only the +artefact each one writes has changed from a ZIP to a footer-augmented +executable. + +## Context + +The executable provider's bundle spec +([`task-sdk/docs/bundle-spec.rst`](../../task-sdk/docs/bundle-spec.rst)) +defines a deployment artifact as a ZIP archive containing: + +1. `airflow-metadata.yaml` declaring `format_version`, `sdk` (language/version), + `source` (archive-relative path to the DAG source file), `executable` + (archive-relative path to the compiled binary), and `dags` (a mapping of + `dag_id` to `{tasks: [task_id, ...]}`). +2. The primary DAG source file, included verbatim. +3. The compiled native executable, which speaks the coordinator protocol + (`--comm=<addr>` / `--logs=<addr>`). + +Bundle authors today produce the executable with a plain `go build` +(see [`go-sdk/example/bundle/Justfile`](../example/bundle/Justfile)). There is +no SDK-provided way to produce the conforming ZIP, so each author would need +to hand-roll one. + +The bundle binary already exposes a `--bundle-metadata` flag (defined in +[`bundle/bundlev1/bundlev1server/server.go`](../bundle/bundlev1/bundlev1server/server.go)) +that prints the `BundleInfo{Name, Version}` returned by the author's +`BundleProvider.GetBundleVersion()`. It does **not** currently invoke +`RegisterDags`, so it does not yet enumerate `dag_id` / `task_id` for the +manifest. This is relevant context: the binary itself is the authoritative +source of dag/task identity at runtime, and the SDK can extend the +introspection path cheaply. + +The user's initial framing was `go build -toolexec`. `-toolexec` wraps each +toolchain invocation (compile, asm, link) and does not have visibility into +the final `-o` output path or a single "build finished" hook, so it is a poor +fit for producing the final ZIP. The options below cover the mechanisms that +do fit, plus the `-toolexec` variant for completeness. + +A packing mechanism has two sub-decisions: + +- **Where the packing logic runs.** In the bundle binary itself + (self-pack), in a separate SDK CLI, or in build tooling outside the SDK + (Makefile/Justfile snippet). +- **How dag/task IDs reach the manifest.** Runtime introspection of the + built binary (call into `RegisterDags` against an in-memory + registry recorder), static AST scan of the source file, or + hand-written manifest. + +The options below combine those two sub-decisions in different ways. + +## Options + +### Option A: Standalone SDK packer CLI (`airflow-go-pack`) + +A new binary under `go-sdk/cmd/airflow-go-pack` that takes +already-built inputs and writes the ZIP: + +``` +airflow-go-pack \ + --source ./example/bundle/main.go \ + --executable ./bin/example-dag-bundle \ + --output ./bin/example.zip +``` + +Manifest population: the packer execs the supplied executable with +`--bundle-metadata` and reads the JSON from stdout to fill `sdk.version`, +and a new `--dump-dags` (or extended `--bundle-metadata`) flag to enumerate +`dags`. Source language is hard-coded to `go`; SDK version is read from the +build info embedded in the binary or from a build-time `-ldflags` value. + +- **Pros:** simple, single-purpose binary; works against any binary the user + built however they like (CGO, cross-compile, custom `-ldflags`); no + coupling to `go build` invocation; trivially callable from `just`, + `make`, CI, or `go generate`. +- **Cons:** two-step UX (`go build` then `airflow-go-pack`); user has to + install or `go run` the tool; nothing prevents pack/build mismatch + (e.g. packing yesterday's binary). + +### Option B: All-in-one SDK CLI with a `build` subcommand + +A single SDK CLI (`airflow-go`) with subcommands that wrap `go build` and +then pack: + +``` +airflow-go build ./example/bundle --output ./bin/example.zip +``` + +Internally: spawn `go build -o <tmp>/bundle <pkg>`, then run the same +introspection step as Option A, then write the ZIP. + +- **Pros:** single command; no chance of pack/build skew; easy to add + related subcommands later (`airflow-go new`, `airflow-go run`, + `airflow-go validate`); good defaults for `-ldflags` (e.g. + `-X main.bundleVersion=...`) without the author having to know them. +- **Cons:** the SDK now owns a `go build` wrapper and inherits + responsibility for forwarding the long tail of `go build` flags + (`-tags`, `-trimpath`, `GOOS`/`GOARCH` env, `-ldflags` passthrough, + `-buildvcs`, etc.); harder to integrate with non-trivial existing build + systems that already drive `go build` themselves. + +### Option C: Self-packing binary (`--pack-bundle <out.zip>`) + +Extend `bundlev1server.Serve` so that when the binary is invoked with +`--pack-bundle <out.zip>`, it builds the ZIP itself: it knows its own +executable path (`os.Executable()`), its embedded source (via `//go:embed` +of the DAG source file at build time), and its dag/task list (by +calling `RegisterDags` against an in-memory recorder). After writing +the archive, it exits. + +- **Pros:** zero extra tools; the binary is fully self-describing; pack + output is provably consistent with the binary's runtime behaviour. +- **Cons:** requires the author's `main` package to embed its own source + (`//go:embed main.go` or similar), which is awkward when the DAG is + spread across multiple files or the source path is non-obvious; + bloats every bundle binary with packing code and an embedded copy of + the source; mixes build-time concerns into a runtime entrypoint. + +### Option D: Two-phase external introspection (introspection binary + packer) + +Same shape as Option A or B, but standardise the introspection contract: +the SDK guarantees that every bundle binary supports +`--dump-bundle-spec` (or a richer `--bundle-metadata`) which prints a +JSON blob containing `sdk.language`, `sdk.version`, and the full `dags` +mapping. The packer's only job is to combine that JSON, the source +file path the user passes in, and the binary itself into a ZIP. + +This is really a refinement of A/B that fixes the introspection contract +in the SDK protocol, rather than an independent option, but is worth +calling out because the shape of the introspection flag is itself a +decision (single flag vs. several; JSON vs. YAML; pretty vs. compact). + +- **Pros:** decouples "how do we enumerate dags" from "how do we ZIP"; + any future packer (third-party CI plugin, IDE, etc.) can rely on the + same contract; trivial to unit-test. +- **Cons:** locks in a wire format the SDK has to keep stable; slightly + more code in the bundle binary than today. + +### Option E: Static AST scan, no introspection + +Parser-only packer: walk the DAG source AST, find `dagbag.AddDag("X")` +calls and the `.AddTask(fn)` calls chained off them, and synthesise the +manifest without running the binary. + +- **Pros:** no runtime dependency on the binary (works even if it + doesn't build for the host platform, e.g. cross-compiled for Linux on Review Comment: Option E's pro is *"works even if the binary doesn't build for the host platform, e.g. cross-compiled for Linux on a macOS dev box."* The chosen Option D requires execing the freshly built binary with `--dump-bundle-spec`, which doesn't work for cross-compiles. ADR 0002 then makes that exec part of the default packer flow. So how does cross-compile work in practice? Build on `darwin/arm64` for `GOOS=linux GOARCH=amd64`, then `go tool airflow-go-pack` execs an x86_64 ELF on darwin and fails with `exec format error`. Either Option D needs Rosetta/qemu as a documented assumption, or the chosen mechanism doesn't actually cover the deploy-from-laptop workflow that's the implicit target use case. Worth either calling this limitation out explicitly in the Consequences section, or sketching how the packer detects cross-compile and falls back. ########## go-sdk/adr/0003-coordinator-protocol-msgpack-ipc.md: ########## @@ -0,0 +1,381 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 3. Dual-mode bundle binary: msgpack-over-IPC coordinator protocol alongside the existing go-plugin/Edge-Worker path + +Date: 2026-04-30 + +## Status + +Accepted. + +## Context + +A Go SDK bundle binary today (the artefact built from +[`go-sdk/example/bundle/main.go`](../example/bundle/main.go) via +`bundlev1server.Serve`) speaks exactly one protocol: HashiCorp +[`go-plugin`](https://github.com/hashicorp/go-plugin) gRPC over a +stdio-negotiated socket, gated by the magic-cookie handshake declared in +[`pkg/bundles/shared/handshake.go`](../pkg/bundles/shared/handshake.go). +The Airflow Go *Edge Worker* +([`cmd/airflow-go-edge-worker`](../cmd/airflow-go-edge-worker/main.go), +[`edge/`](../edge)) is the consumer of that protocol — it execs the +bundle binary as a child process, completes the go-plugin handshake, +opens the `DagBundle` gRPC client, and drives `GetMetadata`/`Execute` +([`bundle/bundlev1/bundlev1server/impl/plugin.go`](../bundle/bundlev1/bundlev1server/impl/plugin.go)). +The bundle binary never listens on a public socket; the protocol is +local-process only. + +Meanwhile, the Python side of Airflow has standardised on a different +wire protocol for non-Python language runtimes — the *coordinator +protocol* — pioneered by the Java SDK and described in +[java-sdk ADR 0002](../../java-sdk/adr/0002-dag-parsing.md) +and +[java-sdk ADR 0003](../../java-sdk/adr/0003-workload-execution.md). +Its shape is: + +- The runtime is launched with `--comm=<host:port>` and + `--logs=<host:port>` CLI arguments. +- It connects out (TCP, loopback) to both addresses. +- Frames on the comm channel are length-prefixed msgpack: a 4-byte + big-endian length followed by the msgpack payload. Requests are + `[id, body]`; responses are `[id, body, error]`. +- Two workloads share one channel, distinguished by the first inbound + frame: `DagFileParseRequest` (one-shot, returns + `DagFileParsingResult` and exits) or `StartupDetails` (multi-round + task execution: the runtime sends `GetConnection` / `GetVariable` / + `GetXCom` / `SetXCom` and terminates with `SucceedTask` or + `TaskState`). +- The logs channel carries structured JSON log records emitted by the + runtime. + +The Python-side launcher is +[`ExecutableCoordinator`](../../task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py), +which already builds command lines of the form +`<binary> --comm=<addr> --logs=<addr>` for both `dag_parsing_runtime_cmd` +and `task_execution_runtime_cmd`. The bundle-spec contract +([`task-sdk/docs/bundle-spec.rst`](../../task-sdk/docs/bundle-spec.rst)) +ratifies that any compiled SDK shipping a ZIP bundle "MUST honour the +SDK coordinator protocol (`--comm=<addr>` / `--logs=<addr>` +socket-based IPC)". The Java SDK satisfies this contract; the Go SDK +currently does not. + +The two protocols target different deployment shapes: + +- **go-plugin / Edge Worker.** The Go-native worker is itself a long-running + process that loads bundles in-process and dispatches tasks to them + over gRPC. It is the only consumer that speaks go-plugin to a Go + bundle today, and it owns the full task-runtime stack on the worker + host (no Python in the data path). This is the path + [`go-sdk/example/bundle/main.go`](../example/bundle/main.go) was + written for and the path that + [`pkg/worker`](../pkg/worker) drives. +- **Coordinator / `ExecutableCoordinator`.** The Python task + runner forks a child that runs `<binary> --comm=… --logs=…`, + bridges its socket to the Airflow supervisor's fd 0, and proxies + Airflow service calls (`GetConnection`, `GetVariable`, ...) through + to the Execution API. This is how Airflow runs non-Python tasks + *without* a per-language worker — the same way Java runs today, and + the same way Rust/C++/Zig will run in the future. It is also the + only path the executable provider's bundle spec recognises. + +Today these two paths require two different binaries, even though the +DAG/task definitions, the registry, the worker plumbing, and the +serialisation surfaces overlap almost entirely. That is the gap this +ADR closes. + +The user-written `main()` is one line — +`bundlev1server.Serve(&myBundle{})` — and we want to keep it one line. +Whichever protocol the binary should speak must be decided inside +`Serve` based on how it was invoked, not by branching in user code. + +## Decision + +Make the SDK bundle binary **dual-mode**. A single +`bundlev1server.Serve(bundle, opts...)` call dispatches to one of two +protocol servers based on its CLI arguments and process environment. +User code does not change. + +### Invocation matrix + +`Serve` parses flags first, then chooses a mode in this order: + +| Trigger | Mode | Behaviour | +|--------------------------------------------------------|-----------------|-----------| +| `--bundle-metadata` | metadata-dump | Existing flag (ADR 0001 / `server.go:37`). Prints `BundleInfo` JSON and exits. | +| `--dump-bundle-spec` | spec-dump | Existing flag added by [ADR 0002](0002-use-go-tool-directive-for-bundle-packer.md). Prints the full bundle spec JSON (`sdk`, `dags`) used by `airflow-go-pack`. | +| `--comm=<host:port> --logs=<host:port>` | **coordinator** | New. Speaks the msgpack-over-IPC coordinator protocol. Both flags are required; partial use is a hard error. | +| `AIRFLOW_BUNDLE_MAGIC_COOKIE` env var present (default) | go-plugin | Existing behaviour. Hands off to `plugin.Serve` which performs the handshake and serves `DagBundle` gRPC to the Edge Worker. | +| Otherwise | error | Print usage to stderr and exit non-zero. Today this case implicitly errors via go-plugin's failed handshake; we make the diagnostic explicit so authors running the binary directly get a clear message. | + +The two server modes share the same `bundlev1.BundleProvider` +implementation and the same lazy `RegisterDags` recorder cache that +`impl.server` already maintains (`impl/plugin.go:99-121`). Only the +front door changes. + +### Coordinator mode: protocol details + +When `Serve` enters coordinator mode it: + +1. **Parses and validates the addresses.** Both `--comm` and `--logs` + are `host:port` strings. `127.0.0.1` is the only host the coordinator + protocol is designed for, but we do not pin it — the value is whatever + `_runtime_subprocess_entrypoint` chose on the Python side. + +2. **Connects out** to the comm address, then to the logs address. Both + are TCP. We dial; we do not listen. The launcher already has both + listeners up before exec'ing the binary + ([java-sdk ADR 0002, "What the Base Class Handles Automatically"](../../java-sdk/adr/0002-dag-parsing.md#what-the-base-class-handles-automatically)). + +3. **Routes structured logs to the logs socket.** A new + `slog.Handler` writes JSON-line records (one record per line, UTF-8, + newline-terminated) to the logs connection, replacing the + `hclog`/stderr handler used in go-plugin mode. `slog.SetDefault` is + called before any user code runs so `log` arguments injected into + tasks land on the right channel. On disconnect the handler falls + back to stderr so the binary never deadlocks on a closed sink. + +4. **Reads the first comm frame and dispatches by message type.** The + first frame's body has a `type` field per the Java SDK's encoding + ([java-sdk ADR 0003, "Task SDK Protocol Messages"](../../java-sdk/adr/0003-workload-execution.md#task-sdk-protocol-messages)). + Two values are valid here: + + - `DagFileParseRequest` → DAG-parsing one-shot. + - `StartupDetails` → task execution. + + Any other type is an error frame back to the supervisor and + `os.Exit(1)`. + +#### DAG-parsing path (`DagFileParseRequest` → `DagFileParsingResult`) + +```text +Supervisor Bundle binary (Go) + │ │ + ├── [4B len][msgpack: id, ─────────────►│ + │ {type: "DagFileParseRequest", │ + │ file: "<bundle path>"}] │ + │ │ + │ ├── BundleProvider.RegisterDags(reg) + │ │ (cached, same as gRPC path) + │ │ + │ ├── serialise(reg) → + │ │ DagFileParsingResult + │ │ in DagSerialization v3 JSON + │ │ (see java-sdk ADR-0002) + │ │ + │◄────────────────[4B len][msgpack: ────┤ + │ id, {type: "DagFileParsingResult", + │ fileloc: "...", + │ serialized_dags: [...] }] │ + │ │ + │ └── close + exit(0) +``` + +The serialised DAG payload must match Python's `SerializedDAG.serialize_dag` +output **exactly**, including the `__type` / `__var` wrapping rules, +unwrapping of "non-decorated" fields (`start_date`, `end_date`, `tags`), +and the timetable encoding listed in +[java-sdk ADR 0002, "DagFileParsingResult Format"](../../java-sdk/adr/0002-dag-parsing.md#dagfileparsingresult-format). +The Go SDK gains a `serde` package that performs this encoding from +`bundlev1.Bundle` / `bundlev1.Task`, validated against +`validation/serialization/test_dags.yaml` (the same fixture set the Java +SDK uses), so the Go and Java outputs are byte-identical for shared +inputs. + +#### Task-execution path (`StartupDetails` → multi-round → `SucceedTask` / `TaskState`) + +```text +Supervisor Bundle binary (Go) + │ │ + ├── StartupDetails ────────────────────►│ + │ (ti, dag_rel_path, bundle_info, │ + │ start_date, ti_context) │ + │ │ + │ ├── lookup task: + │ │ bundle.dags[ti.dag_id] + │ │ .tasks[ti.task_id] + │ │ (returns TaskState{state:"removed"} + │ │ if not found, mirroring Java) + │ │ + │ ├── construct sdk.Client whose + │ │ GetConnection / GetVariable / + │ │ GetXCom / SetXCom calls block on + │ │ request/response over the + │ │ comm socket + │ │ + │◄── GetConnection(conn_id) ────────────┤ + ├── ConnectionResult ──────────────────►│ + │◄── GetVariable(key) ──────────────────┤ + ├── VariableResult ────────────────────►│ + │◄── GetXCom(...) ──────────────────────┤ + ├── XComResult ────────────────────────►│ + │◄── SetXCom(...) ──────────────────────┤ + ├── (empty response) ──────────────────►│ + │ │ + │ ├── task fn returns: + │ │ err == nil → SucceedTask + │ │ err != nil → TaskState{"failed"} + │ │ (panic recovered → "failed") Review Comment: `(panic recovered -> failed)` is a new task-runtime semantic. Existing go-plugin mode via `pkg/worker.Worker.ExecuteTaskWorkload` doesn't appear to recover panics today -- they propagate through gRPC to the Edge Worker, which marks the task failed via its own crash detection. If coordinator mode adds a `recover()` while go-plugin mode doesn't, the same user task code panicking gets different observability and failure attribution depending on which runtime picks it up (coordinator: clean failure record; go-plugin: process crash trace in Edge Worker logs). Either lift the recover into shared `pkg/worker` so both modes match, or document the divergence in this ADR so future debugging knows to expect it. ########## go-sdk/adr/0003-coordinator-protocol-msgpack-ipc.md: ########## @@ -0,0 +1,381 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 3. Dual-mode bundle binary: msgpack-over-IPC coordinator protocol alongside the existing go-plugin/Edge-Worker path + +Date: 2026-04-30 + +## Status + +Accepted. + +## Context + +A Go SDK bundle binary today (the artefact built from +[`go-sdk/example/bundle/main.go`](../example/bundle/main.go) via +`bundlev1server.Serve`) speaks exactly one protocol: HashiCorp +[`go-plugin`](https://github.com/hashicorp/go-plugin) gRPC over a +stdio-negotiated socket, gated by the magic-cookie handshake declared in +[`pkg/bundles/shared/handshake.go`](../pkg/bundles/shared/handshake.go). +The Airflow Go *Edge Worker* +([`cmd/airflow-go-edge-worker`](../cmd/airflow-go-edge-worker/main.go), +[`edge/`](../edge)) is the consumer of that protocol — it execs the +bundle binary as a child process, completes the go-plugin handshake, +opens the `DagBundle` gRPC client, and drives `GetMetadata`/`Execute` +([`bundle/bundlev1/bundlev1server/impl/plugin.go`](../bundle/bundlev1/bundlev1server/impl/plugin.go)). +The bundle binary never listens on a public socket; the protocol is +local-process only. + +Meanwhile, the Python side of Airflow has standardised on a different +wire protocol for non-Python language runtimes — the *coordinator +protocol* — pioneered by the Java SDK and described in +[java-sdk ADR 0002](../../java-sdk/adr/0002-dag-parsing.md) +and +[java-sdk ADR 0003](../../java-sdk/adr/0003-workload-execution.md). +Its shape is: + +- The runtime is launched with `--comm=<host:port>` and + `--logs=<host:port>` CLI arguments. +- It connects out (TCP, loopback) to both addresses. +- Frames on the comm channel are length-prefixed msgpack: a 4-byte + big-endian length followed by the msgpack payload. Requests are + `[id, body]`; responses are `[id, body, error]`. +- Two workloads share one channel, distinguished by the first inbound + frame: `DagFileParseRequest` (one-shot, returns + `DagFileParsingResult` and exits) or `StartupDetails` (multi-round + task execution: the runtime sends `GetConnection` / `GetVariable` / + `GetXCom` / `SetXCom` and terminates with `SucceedTask` or + `TaskState`). +- The logs channel carries structured JSON log records emitted by the + runtime. + +The Python-side launcher is +[`ExecutableCoordinator`](../../task-sdk/src/airflow/sdk/coordinators/executable/coordinator.py), +which already builds command lines of the form +`<binary> --comm=<addr> --logs=<addr>` for both `dag_parsing_runtime_cmd` +and `task_execution_runtime_cmd`. The bundle-spec contract +([`task-sdk/docs/bundle-spec.rst`](../../task-sdk/docs/bundle-spec.rst)) +ratifies that any compiled SDK shipping a ZIP bundle "MUST honour the +SDK coordinator protocol (`--comm=<addr>` / `--logs=<addr>` +socket-based IPC)". The Java SDK satisfies this contract; the Go SDK +currently does not. + +The two protocols target different deployment shapes: + +- **go-plugin / Edge Worker.** The Go-native worker is itself a long-running + process that loads bundles in-process and dispatches tasks to them + over gRPC. It is the only consumer that speaks go-plugin to a Go + bundle today, and it owns the full task-runtime stack on the worker + host (no Python in the data path). This is the path + [`go-sdk/example/bundle/main.go`](../example/bundle/main.go) was + written for and the path that + [`pkg/worker`](../pkg/worker) drives. +- **Coordinator / `ExecutableCoordinator`.** The Python task + runner forks a child that runs `<binary> --comm=… --logs=…`, + bridges its socket to the Airflow supervisor's fd 0, and proxies + Airflow service calls (`GetConnection`, `GetVariable`, ...) through + to the Execution API. This is how Airflow runs non-Python tasks + *without* a per-language worker — the same way Java runs today, and + the same way Rust/C++/Zig will run in the future. It is also the + only path the executable provider's bundle spec recognises. + +Today these two paths require two different binaries, even though the +DAG/task definitions, the registry, the worker plumbing, and the +serialisation surfaces overlap almost entirely. That is the gap this +ADR closes. + +The user-written `main()` is one line — +`bundlev1server.Serve(&myBundle{})` — and we want to keep it one line. +Whichever protocol the binary should speak must be decided inside +`Serve` based on how it was invoked, not by branching in user code. + +## Decision + +Make the SDK bundle binary **dual-mode**. A single +`bundlev1server.Serve(bundle, opts...)` call dispatches to one of two +protocol servers based on its CLI arguments and process environment. +User code does not change. + +### Invocation matrix + +`Serve` parses flags first, then chooses a mode in this order: + +| Trigger | Mode | Behaviour | +|--------------------------------------------------------|-----------------|-----------| +| `--bundle-metadata` | metadata-dump | Existing flag (ADR 0001 / `server.go:37`). Prints `BundleInfo` JSON and exits. | +| `--dump-bundle-spec` | spec-dump | Existing flag added by [ADR 0002](0002-use-go-tool-directive-for-bundle-packer.md). Prints the full bundle spec JSON (`sdk`, `dags`) used by `airflow-go-pack`. | +| `--comm=<host:port> --logs=<host:port>` | **coordinator** | New. Speaks the msgpack-over-IPC coordinator protocol. Both flags are required; partial use is a hard error. | +| `AIRFLOW_BUNDLE_MAGIC_COOKIE` env var present (default) | go-plugin | Existing behaviour. Hands off to `plugin.Serve` which performs the handshake and serves `DagBundle` gRPC to the Edge Worker. | +| Otherwise | error | Print usage to stderr and exit non-zero. Today this case implicitly errors via go-plugin's failed handshake; we make the diagnostic explicit so authors running the binary directly get a clear message. | + +The two server modes share the same `bundlev1.BundleProvider` +implementation and the same lazy `RegisterDags` recorder cache that +`impl.server` already maintains (`impl/plugin.go:99-121`). Only the +front door changes. + +### Coordinator mode: protocol details + +When `Serve` enters coordinator mode it: + +1. **Parses and validates the addresses.** Both `--comm` and `--logs` + are `host:port` strings. `127.0.0.1` is the only host the coordinator + protocol is designed for, but we do not pin it — the value is whatever + `_runtime_subprocess_entrypoint` chose on the Python side. + +2. **Connects out** to the comm address, then to the logs address. Both + are TCP. We dial; we do not listen. The launcher already has both + listeners up before exec'ing the binary + ([java-sdk ADR 0002, "What the Base Class Handles Automatically"](../../java-sdk/adr/0002-dag-parsing.md#what-the-base-class-handles-automatically)). + +3. **Routes structured logs to the logs socket.** A new + `slog.Handler` writes JSON-line records (one record per line, UTF-8, + newline-terminated) to the logs connection, replacing the + `hclog`/stderr handler used in go-plugin mode. `slog.SetDefault` is + called before any user code runs so `log` arguments injected into + tasks land on the right channel. On disconnect the handler falls + back to stderr so the binary never deadlocks on a closed sink. + +4. **Reads the first comm frame and dispatches by message type.** The + first frame's body has a `type` field per the Java SDK's encoding + ([java-sdk ADR 0003, "Task SDK Protocol Messages"](../../java-sdk/adr/0003-workload-execution.md#task-sdk-protocol-messages)). + Two values are valid here: + + - `DagFileParseRequest` → DAG-parsing one-shot. + - `StartupDetails` → task execution. + + Any other type is an error frame back to the supervisor and + `os.Exit(1)`. + +#### DAG-parsing path (`DagFileParseRequest` → `DagFileParsingResult`) + +```text +Supervisor Bundle binary (Go) + │ │ + ├── [4B len][msgpack: id, ─────────────►│ + │ {type: "DagFileParseRequest", │ + │ file: "<bundle path>"}] │ + │ │ + │ ├── BundleProvider.RegisterDags(reg) + │ │ (cached, same as gRPC path) + │ │ + │ ├── serialise(reg) → + │ │ DagFileParsingResult + │ │ in DagSerialization v3 JSON + │ │ (see java-sdk ADR-0002) + │ │ + │◄────────────────[4B len][msgpack: ────┤ + │ id, {type: "DagFileParsingResult", + │ fileloc: "...", + │ serialized_dags: [...] }] │ + │ │ + │ └── close + exit(0) +``` + +The serialised DAG payload must match Python's `SerializedDAG.serialize_dag` +output **exactly**, including the `__type` / `__var` wrapping rules, +unwrapping of "non-decorated" fields (`start_date`, `end_date`, `tags`), +and the timetable encoding listed in +[java-sdk ADR 0002, "DagFileParsingResult Format"](../../java-sdk/adr/0002-dag-parsing.md#dagfileparsingresult-format). +The Go SDK gains a `serde` package that performs this encoding from +`bundlev1.Bundle` / `bundlev1.Task`, validated against +`validation/serialization/test_dags.yaml` (the same fixture set the Java +SDK uses), so the Go and Java outputs are byte-identical for shared +inputs. + +#### Task-execution path (`StartupDetails` → multi-round → `SucceedTask` / `TaskState`) + +```text +Supervisor Bundle binary (Go) + │ │ + ├── StartupDetails ────────────────────►│ + │ (ti, dag_rel_path, bundle_info, │ + │ start_date, ti_context) │ + │ │ + │ ├── lookup task: + │ │ bundle.dags[ti.dag_id] + │ │ .tasks[ti.task_id] + │ │ (returns TaskState{state:"removed"} + │ │ if not found, mirroring Java) + │ │ + │ ├── construct sdk.Client whose + │ │ GetConnection / GetVariable / + │ │ GetXCom / SetXCom calls block on + │ │ request/response over the + │ │ comm socket + │ │ + │◄── GetConnection(conn_id) ────────────┤ + ├── ConnectionResult ──────────────────►│ + │◄── GetVariable(key) ──────────────────┤ + ├── VariableResult ────────────────────►│ + │◄── GetXCom(...) ──────────────────────┤ + ├── XComResult ────────────────────────►│ + │◄── SetXCom(...) ──────────────────────┤ + ├── (empty response) ──────────────────►│ + │ │ + │ ├── task fn returns: + │ │ err == nil → SucceedTask + │ │ err != nil → TaskState{"failed"} + │ │ (panic recovered → "failed") + │ │ + │◄── SucceedTask / TaskState ───────────┤ + │ │ + │ └── close + exit(0) +``` + +Concretely, this reuses +[`pkg/worker.Worker`](../pkg/worker/runner.go) for task lookup and +parameter injection — `extract(ctx, sdk.Client, *slog.Logger)`, +`transform(ctx, sdk.VariableClient, *slog.Logger)`, and `load() error` +in the example bundle work unchanged. The injected `sdk.Client` +implementation is swapped: in go-plugin mode it talks to the Execution +API directly via the URL from viper (`impl/plugin.go:182`), in +coordinator mode it talks to the supervisor over the comm socket. +Both implement the same `sdk.Client` / `sdk.VariableClient` interfaces, +so user task code is identical between the two modes. + +Frame correlation, error envelopes, and request `id` numbering follow +java-sdk ADR 0003 verbatim. Re-implementing rather than reusing those +is a deliberate cost of having a separate Go runtime; the validation +fixtures keep the encoders honest. + +### go-plugin mode: unchanged + +When neither dump flag nor `--comm`/`--logs` is set, `Serve` falls +through to the existing call site: + +```go +plugin.Serve(&plugin.ServeConfig{ + HandshakeConfig: shared.Handshake, + Plugins: plugin.PluginSet{"dag-bundle": &impl.BundleGRPCPlugin{...}}, + GRPCServer: plugin.DefaultGRPCServer, +}) +``` + +The handshake env var (`AIRFLOW_BUNDLE_MAGIC_COOKIE`) gates the path +the same way it does today, so an Edge Worker that execs the binary +gets exactly the same protocol it gets today. The `DagBundle` gRPC +service, the registry cache, the `--bundle-metadata` flag, and the +worker injection in +[`impl/plugin.go:178`](../bundle/bundlev1/bundlev1server/impl/plugin.go) +are untouched. + +### Code organisation + +A new internal package +`go-sdk/bundle/bundlev1/bundlev1server/impl/coord` owns the +coordinator-mode server: frame codec, log-sink handler, dag-parse +handler, task-execution handler, and the `sdk.Client` adapter that +proxies to the comm socket. It depends on a new +`go-sdk/bundle/bundlev1/serde` package for DagSerialization v3 +encoding. The frame codec is small enough to keep first-party rather +than pulling a new msgpack dependency at the API surface; we use +[`github.com/vmihailenco/msgpack/v5`](https://github.com/vmihailenco/msgpack) +internally. + +`bundlev1server.Serve` becomes: + +```go +func Serve(bundle bundlev1.BundleProvider, opts ...ServeOpt) error { + config.SetupViper("") + flag.Parse() + + switch mode := decideMode(); mode { + case modeMetadataDump: + return dumpBundleMetadata(bundle) // existing + case modeSpecDump: + return dumpBundleSpec(bundle) // ADR 0002 + case modeCoordinator: + return coord.Serve(bundle, *commAddr, *logsAddr) // NEW + case modePlugin: + return servePlugin(bundle) // existing + } +} +``` + +User code (`main.go`) is the same one line: + +```go +func main() { bundlev1server.Serve(&myBundle{}) } +``` + +## Consequences + +### Capability gains + +- A single binary built from one `bundlev1server.Serve` entry point now + runs under both the Go-native Edge Worker (go-plugin) and the + Python-native task runner via `ExecutableCoordinator` + (msgpack-over-IPC). Authors do not pick a deployment shape at build + time. +- The bundle ZIP produced by `airflow-go-pack` (ADR 0002) becomes + spec-conformant Review Comment: *"the bundle ZIP produced by `airflow-go-pack`"* -- ADR 0004 supersedes the ZIP container with the self-contained executable, but the Status section at the top of this ADR doesn't mention that supersession the way ADRs 0001 and 0002 do. Readers landing here without ADR 0004 context will hit the stale "ZIP" reference and end up reasoning about a wrong artefact shape. Either swap "ZIP" for "bundle file" / "bundle artefact", or add a `Superseded in part by ADR 0004` note in Status. ########## go-sdk/adr/0004-self-contained-executable-bundle.md: ########## @@ -0,0 +1,304 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 4. Self-contained executable bundle (footer-embedded source and metadata) + +Date: 2026-05-04 + +## Status + +Accepted. Supersedes the ZIP-archive container portion of +[ADR 0001](0001-bundle-packing-options.md) and the ZIP output sketched +in [ADR 0002](0002-use-go-tool-directive-for-bundle-packer.md). The +packer mechanism (Option A standalone packer + Option D introspection +contract + Option H `tool` directive) is unchanged; only the artefact +the packer writes is changed. + +## Context + +ADR 0001 / ADR 0002 picked a ZIP archive as the bundle container, +following the executable provider's existing +[`task-sdk/docs/bundle-spec.rst`](../../task-sdk/docs/bundle-spec.rst). +A conforming bundle today is `bundle.zip` with three required entries: +`airflow-metadata.yaml`, the primary DAG source file, and the compiled +executable. + +That layout has three properties we want to preserve: + +1. **Discovery without execution.** The scanner must be able to read + `dag_id` / `task_id` and the SDK language/version from a bundle on + disk without running the binary. ADR 0002 already enforces this — + `airflow-go-pack` runs the binary once at build time, captures its + `--dump-bundle-spec` output into the manifest, and the scanner reads + the manifest at deploy time. +2. **Source available for the UI.** The Airflow UI's source-view + panel needs to render the DAG file. The current spec ships it as a + verbatim ZIP entry referenced by the manifest's `source` field. +3. **Single deployment unit.** Drop one file in + `[executable] bundles_folder` and the scanner picks it up. + +What the ZIP container costs us: + +- **Two artefacts in flight.** `go build` produces a binary; the + packer wraps it into a ZIP. Anything that touches the binary after + it is wrapped (re-strip, re-sign, swap-in a debug build) drifts from + the manifest unless the wrapping is redone. The wrapping step is + cheap but the drift mode is real. +- **A second container format on the consumer side.** The scanner + must open archives, find members by name, and materialise the + executable into a transient cache before the runtime can exec it. + That is `archive/zip` on the Python side plus a per-bundle cache + directory. +- **Inspection requires a different tool than running.** `unzip` to + inspect, then run; or run, then `unzip` to debug. Two muscle memories. + +Native-executable SDKs (Go, Rust, C++, Zig) all produce a single +self-contained binary by static linking. The binary itself is already +the only thing that has to land on the worker host to run a task. The +manifest and the source file are small data the scanner needs but the +runtime doesn't. Both can ride along in a footer appended to the +binary, with the binary remaining a runnable executable. + +This is the same pattern self-extracting installers, `goreleaser`-style +self-update images, and embedded-asset binaries already use: append +data after the OS-recognised binary structure, leave a fixed-size +trailer at the very end so a reader can locate the data, and validate +with a magic value. + +The user-facing claim becomes "the executable *is* the bundle." A +bundle directory looks like: + +``` +/opt/airflow/executable-bundles/ +├── example +├── pipeline +└── analytics +``` + +(Filenames follow OS conventions: no extension on Linux/macOS, `.exe` +on Windows. The scanner identifies bundles by the trailer's magic, not +by the filename.) + +## Decision + +Replace the bundle's ZIP container with a footer appended to the +compiled executable. The executable's normal byte content is unchanged +and it remains directly runnable; the footer is data that follows the +last byte the OS loader cares about. + +### Footer layout + +A bundle file is laid out as: + +```text ++---------------------------------+ +| <native executable: ELF/Mach-O/PE, | +| including any code-signing structures> | ++---------------------------------+ <- end of "binary" region +| source bytes (variable length) | raw root source file, UTF-8, +| | length = source_len; MAY be 0 ++---------------------------------+ +| metadata bytes (variable length)| airflow-metadata.yaml content, +| | UTF-8, length = metadata_len ++---------------------------------+ +| trailer (32 bytes, little-endian fixed layout): | +| bytes 0..3 source_len u32 | +| bytes 4..7 metadata_len u32 | +| bytes 8..11 footer_ver u32 (= 1) | +| bytes 12..23 reserved 12 bytes, zero | +| bytes 24..31 magic 8 bytes ASCII "AFBNDL01" | ++---------------------------------+ <- EOF +``` + +`AFBNDL01` is `0x41 0x46 0x42 0x4E 0x44 0x4C 0x30 0x31`. The two +trailing ASCII digits are the footer-format version, repeated for human +inspection (`tail -c 8 ./mybundle | xxd`); the binary `footer_ver` +field is the source of truth for parsing. + +Reader algorithm: + +1. Open the file. Seek to `EOF - 32`. Read 32 bytes. +2. Compare bytes 24..31 against `AFBNDL01`. If different, the file is + not a bundle; the scanner ignores it. +3. Parse `footer_ver`. If unknown, fail with a versioning error. +4. Compute `metadata_start = filesize - 32 - metadata_len` and + `source_start = metadata_start - source_len`. +5. Read `metadata_len` bytes from `metadata_start` for the manifest. +6. Read `source_len` bytes from `source_start` for the source view. + If `source_len == 0`, no source is embedded; the UI falls back to + "(source not available)". +7. Validate that `source_start >= 0` and that the implied "binary + region" (bytes `[0, source_start)`) is non-empty. + +Ordering note: source comes *before* metadata so a future +`format_version` can introduce extra trailing blobs (e.g. signed +checksums, compressed deps) by extending the trailer rather than +inserting between existing blobs. + +### Manifest schema changes + +The manifest content is the same YAML as today, with two field-level +changes that follow from the footer container: + +- **Drop `executable`.** The binary *is* the file; there is no + archive-relative path to record. +- **Redefine `source` as a display filename, not a path.** The source + bytes live in the footer; the manifest's `source` field carries the + original filename (e.g. `example.go`) so the UI can show it as a + filename in the source-view panel and pick a syntax-highlighting + mode from the extension. + +Everything else (`format_version`, `sdk.language`, `sdk.version`, +`dags`, the open-additivity rule for unknown keys) is unchanged. + +### Build pipeline + +The packer's behaviour from ADR 0002 changes only at the final write +step: + +1. Resolve target package, locate the file with `func main()`. (No + change.) +2. Run `go build [forwarded flags] -o <out> <pkg>`. (No change.) +3. Exec the freshly built binary with `--dump-bundle-spec` to obtain + the manifest. (No change.) +4. **New:** read the source file's bytes; serialise the manifest to + YAML; append `<source><metadata><trailer>` to `<out>`. +5. Default output path becomes `<bundleName>` (or `<bundleName>.exe` + on Windows), not `<bundleName>.zip`. + +Ordering against post-build steps: + +- **Strip:** must run *before* append. Stripping a file that already + has a footer either leaves the footer intact (most strip + implementations stop at the OS-defined end of the binary) or + truncates it; do not rely on either. +- **Code-sign:** must run *after* append on platforms whose signature + covers the entire file (Linux dm-verity, macOS post-Big-Sur for + certain notarisation flows, Windows Authenticode). The signature + then attests to the footer's contents along with the binary, which + is the property we want. +- **Compressors (UPX, etc.):** unsupported. UPX rewrites the file end + to end, destroying the trailer. Bundle binaries should not be + compressor-wrapped; this matches typical production deployment + practice. + +Determinism: the footer is byte-identical for byte-identical inputs +(source bytes, manifest YAML, layout), so a deterministic `go build` +plus a deterministic manifest serialisation produces a byte-identical +bundle file. We canonicalise the manifest as sorted-key YAML at write +time to avoid map-order non-determinism on the Go side. + +### Cross-language scope + +The bundle spec is language-agnostic by design. Every native-SDK +language we currently target (Go, Rust, C++, Zig) emits a single +statically-linked native executable; appending a fixed-format footer Review Comment: *"Every native-SDK language we currently target (Go, Rust, C++, Zig) emits a single statically-linked native executable"* overstates portability. Rust and C++ on Linux typically dynamically link against glibc by default; Go is the unusual one (its runtime explicitly statics by default). The footer-after-EOF trick works on ELF/Mach-O/PE regardless of static-vs-dynamic linking (the loader only consumes `PT_LOAD` segments / `LC_SEGMENT` commands / sections, never reads past EOF), so the underlying claim about the technique is fine. Just drop "statically-linked" or qualify as "single-file native executable" so the next reader doesn't push back on a strictly-untrue premise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
