jason810496 commented on code in PR #67153: URL: https://github.com/apache/airflow/pull/67153#discussion_r3265786591
########## go-sdk/adr/0002-use-go-tool-directive-for-bundle-packer.md: ########## @@ -0,0 +1,222 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 2. Use the Go 1.24 `tool` directive to deliver the bundle packer + +Date: 2026-04-30 + +## Status + +Accepted. Selects from the option register in +[ADR 0001](0001-bundle-packing-options.md). + +The output-format portion of this ADR (the packer writes a ZIP archive +following the bundle spec) is superseded by +[ADR 0004](0004-self-contained-executable-bundle.md): the packer now +writes a self-contained executable with an appended footer carrying +the source bytes and the manifest. The packer's *mechanism* (Option +A standalone binary + Option D introspection contract + Option H +`tool` directive) is unchanged. The decision sketches below mention +ZIP output; read them with the ADR 0004 substitution in mind, and +treat ADR 0004 as authoritative wherever the two disagree. + +## Context + +[ADR 0001](0001-bundle-packing-options.md) enumerated nine candidate +mechanisms for producing a conforming bundle ZIP +([`task-sdk/docs/bundle-spec.rst`](../../task-sdk/docs/bundle-spec.rst)) +from a Go SDK build. Two reasons drive the choice: + +1. **The repository already requires Go 1.24.** `go-sdk/go.mod` declares + `go 1.24.0` with `toolchain go1.24.6`, so language features added in + 1.24 are available to every consumer of the SDK by construction. +2. **Contributors expect Go-native workflows.** The Go 1.24 `tool` + directive is the toolchain's native answer to "depend on a + build-time CLI without polluting the global PATH." It pins the tool + version per-project in `go.mod`, resolves it through the standard + module cache, and exposes it as `go tool <name>`, with no extra + installer and no per-worktree drift. The same problem on the Python + side led `breeze` to switch to `uvx` in + [ADR 0017](../../dev/breeze/doc/adr/0017-use-uvx-to-run-breeze-from-local-sources.md); + `tool` is the analogous answer here. + +The `tool` directive is a delivery mechanism. It still needs an +underlying implementation. We pair it with two of the implementation +options from ADR 0001, with a UX twist: + +- **Option A (standalone packer):** a single-purpose binary at + `go-sdk/cmd/airflow-go-pack`. It still operates as one process with + a clear input/output contract, but it drives `go build` internally + by default so that the common case is one command: + `go tool airflow-go-pack ./pkg`. Authors who already produce their + own binary can opt out via `--executable <path>` and skip the build + phase. This is closer to Option B's ergonomics than the original + ADR 0001 sketch, but kept inside the standalone-packer shape so the + SDK does not own a fully general `go build` wrapper. +- **Option D (standardised introspection contract):** every bundle + binary supports a `--dump-bundle-spec` flag that prints JSON + containing `sdk.language`, `sdk.version`, and the full `dags` mapping. + The packer execs the freshly built binary with this flag to populate + the manifest, which keeps `RegisterDags` as the single authoritative + source of dag/task identity. + +Options C, E, F, G, and I from ADR 0001 are rejected for the reasons +recorded there. Option B (a full `airflow-go build` wrapping `go +build`) is rejected as a separate top-level command, but its core +ergonomic win (one command for the common case) is folded into the +Option A packer through default behaviour, with a `--` passthrough +convention so authors can forward arbitrary flags to the underlying +`go build` without the SDK having to enumerate them. + +## Decision + +1. **Add `cmd/airflow-go-pack` to the go-sdk module.** Its default + invocation is one command: + + ```sh + go tool airflow-go-pack [./path/to/pkg] [-- <go build flags>...] + ``` + + The single positional argument is the Go package containing the + bundle's `main` package (defaults to `.`, the current directory). + Anything after `--` is forwarded verbatim to the internal + `go build` invocation, so authors keep full control over `-tags`, + `-trimpath`, `-ldflags`, `GOOS`/`GOARCH` (via env), `-buildvcs`, + etc. without the packer having to enumerate them. + + With no further flags, the packer: + + 1. Resolves the target package and locates the file in that + package that defines `func main()`. That file becomes the + manifest's `source` and is copied verbatim into the ZIP. If + `main` is split across multiple files, the packer errors and + asks the author to specify `--source <file>`. + 2. Runs `go build [forwarded flags] -o <tmpdir>/<binname> <pkg>`. + 3. Executes the freshly built binary with `--dump-bundle-spec` to + obtain `sdk.{language,version}` and the `dags` mapping. + 4. Writes a deterministic ZIP next to the working directory at + `<bundleName>.zip`, where `<bundleName>` comes from the + binary's `BundleInfo.Name` (already exposed via + `--bundle-metadata`). + + Optional overrides, for advanced or pre-built workflows: + + - `--source <path>`: override the auto-detected source file. + - `--executable <path>`: skip the internal `go build` and pack a + pre-built binary. Mutually exclusive with `--` build-flag + passthrough. + - `--output <path>`: override the default output ZIP path. + + Examples: + + ```sh + # Common case: build and pack in one command from the package dir. + go tool airflow-go-pack + + # Pack a different package, with extra go build flags. + go tool airflow-go-pack ./cmd/my-bundle -- -trimpath -tags=prod + + # Pack an already-built binary (skips go build). + go tool airflow-go-pack --executable ./build/example --source main.go + ``` + +2. **Extend `bundlev1server.Serve` with `--dump-bundle-spec`.** The + flag prints a JSON document of the form: + + ```json Review Comment: We could define the JSON schema of the `airflow-metadata.yaml` format either in this PR or the next one. It was added in https://github.com/apache/airflow/pull/67161/changes/478edabcd14b71fdbd4596ea4521ca26047a7a5b#diff-82337e6c6586f2c271a1457ae2f45acf563aa7b75ce82a78f163cf78690c91b4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
