The GitHub Actions job "Tests (AMD)" on airflow.git/main has succeeded. Run started by GitHub user potiuk (triggered by potiuk).
Head commit for run: c767af5a47b521bb7689913925e5cc07ceb926da / Jason(Zhe-You) Liu <[email protected]> Go-SDK: ADRs for bundle packing and coordinator-protocol runtime (#67153) * Add ADRs for bundle packing options and Go tool directive implementation * Add ADR for dual-mode bundle binary supporting msgpack-over-IPC coordinator protocol * Rename ExecutableRuntimeCoordinator to ExecutableCoordinator across documentation and codebase * Refactor bundle spec to use self-contained executable with embedded metadata and source * CI: Fix the link to other docs * Reflect self-contain binary spec in ADR 0003 * Resolve cross-compile in packer with two-build introspection The packer's manifest-population step execs the freshly built binary with --dump-bundle-spec. On cross-builds (e.g., darwin/arm64 host packing for linux/amd64) the target binary is not runnable on the host, so a single go-build pipeline cannot introspect it. The same issue applies to --executable, where the user may hand the packer a pre-built cross-target binary that the host cannot exec. Specify in ADR 0002 that the packer treats introspection and target-build as separate concerns. The packer attempts to exec the candidate introspection binary and, on "exec format error", builds a host-arch sidecar from the positional package, execs that for --dump-bundle-spec, and appends the resulting footer to the target artifact (whether that artifact came from the internal go build or from --executable). The Go build cache amortises the sidecar to a link step when host arch is already involved, so there is no overhead when no cross-compile is needed and no host-side runner (Rosetta, qemu-user-static) is required for the cross case. If --executable is given without a positional package and the supplied binary is not host-runnable, the packer errors with a message asking for the source package so the sidecar can be built. Add a one-line note to ADR 0001 Option D's cons pointing to ADR 0002 so the trade-off is visible alongside the other options. * Scope Windows out of v1 for self-contained bundle format The footer-after-EOF layout assumes signing tools hash the entire file. That holds on Linux (dm-verity) and macOS (codesign) but not on Windows: PE Authenticode stores its signature in the certificate table referenced from the Optional Header, and Microsoft's EnableCertPaddingCheck hardening (MS13-098) rejects extra bytes after the signature. Appending the bundle footer in either order relative to signtool produces a binary that strict-mode verification rejects, so the current ADR cannot honestly claim Windows support. Remove the .exe defaults and the Authenticode reference from the build-pipeline and out-of-scope sections, and add Windows as an explicit non-goal for v1 with a note that supporting it would require a different layout (e.g. a PE resource section) tracked separately. The packer should refuse GOOS=windows builds in v1 with a clear error rather than silently producing an unsignable artefact. * Drop "statically-linked" qualifier from native-SDK claim The footer-after-EOF technique works because the OS loader stops at the format-defined end of image (ELF section/segment extents, Mach-O LC_SEGMENT extents), not because the binary is statically linked. Rust and C++ default to dynamic glibc on Linux and remain single-file artefacts that take the footer cleanly. Replace "statically-linked native executable" with "single-file native executable" so the reasoning matches the property the technique actually depends on, and fix the parallel claim in the context paragraph above. * Add binary_sha256 to footer; demote OS code-signing to deployment hardening The previous draft of ADR 0004 leaned on OS code-signing (codesign / Authenticode / dm-verity) for bundle tamper detection. That framing was honest only for macOS, and even there it conflated authenticity (signed by a trusted identity) with integrity (file is byte-identical to what the packer produced). Airflow's threat model treats bundles_folder as Deployment-Manager-controlled, so the bundle format only needs to provide integrity against truncation, partial writes, and naive tampering; authenticity is a deployment-time concern that Deployment Managers can layer on top with whatever signing flow matches their platform. Expand the trailer from 32 to 64 bytes to carry binary_sha256 (32-byte SHA-256 over bytes [0, source_start), the binary region only — the hash field is itself in the trailer and cannot cover itself). The scanner verifies the hash at discovery and caches the result by (path, inode, mtime, size) so the runtime does not re-hash on every exec. Mismatch is treated identically to a failed magic check: the scanner skips the file with a clear error. Reframe the code-sign rule: the bundle format neither requires nor prevents OS code-signing. Deployment Managers who want codesign/rcodesign on macOS or fs-verity/IMA on Linux apply it after the footer append so the signature covers the trailer; those who do not (the common Linux deployment) get tamper detection from the embedded hash alone. Narrow the Windows non-goal accordingly. The footer-after-EOF layout runs fine on PE; only Authenticode code-signing is incompatible because MS13-098 rejects extra bytes past the signature. Windows becomes a supported bundle target again (.exe default output restored); Authenticode-signed Windows bundles remain out of scope for v1 and would require a different layout (PE resource section) tracked separately. Note the small verification-to-execve TOCTOU window under "What this costs"; acceptable for v1 given the threat model. * Replace "fully general" hedge with the actual go-build-flag rule The Option A bullet ended with "kept inside the standalone-packer shape so the SDK does not own a fully general go build wrapper", which describes what the packer is not without saying what it is. The real rule is positive: the packer never interprets go build flags; arbitrary flags pass through verbatim after the -- separator, which keeps the packer's flag surface small and stable as go build evolves. State the rule directly. * Resolve dispatch-matrix overlap between go-plugin and error rows The matrix marked the AIRFLOW_BUNDLE_MAGIC_COOKIE row as "(default)" while keeping an "Otherwise -> error" fallthrough row, which made it unclear which row fires when no cookie is set. The actual rule is ordered match: rows 1-4 each require a specific positive trigger (metadata flag, spec-dump flag, comm/logs pair, magic cookie env var), and row 5 fires only when none of those conditions hold. Drop the misleading "(default)" annotation, number the rows so the ordering is explicit, and reword the fallthrough row to make the precedence visible at a glance. * Note that panic recovery is pkg/worker behaviour, not coordinator-only The task-execution diagram shows "(panic recovered -> 'failed')", which a reviewer could plausibly read as a coordinator-mode invention. Recovery is actually pkg/worker.Worker.runTask's existing defer recover() that calls reportStateFailed on panic (runner.go:295-311); both go-plugin mode and coordinator mode reuse the same Worker, so the behaviour is identical between the two modes. Add a paragraph next to the diagram so the property is read as shared rather than divergent. * Make ADRs up to date * Address phani's comment and static check errors * Rename --bundle-metadata flag as --airflow-metadata Report URL: https://github.com/apache/airflow/actions/runs/26876178711 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
