jason810496 commented on code in PR #72043: URL: https://github.com/apache/airflow/pull/72043#discussion_r4013516672
########## go-sdk/adr/0006-mixed-lang-task-handler-interface.md: ########## @@ -0,0 +1,179 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 6. Bundle registration and Mixed Lang task handlers + +Date: 2026-09-09 + +## Status + +Proposed. Keeps the argument binding shipped in #70209, replaces the provider-and-registry +registration flow around it, and reshapes the task signature, following the design review on #72043. + +## Decision + +1. **A bundle is a value the author builds.** `airflow.Bundle()` returns a `*airflow.BundleRef`; + `main` reads build, register, serve, with `bundle.Serve()` as its last statement. +2. **`bundle.Register(items ...airflow.Registration)`** is the single registration verb, taking + native Dags ([ADR 7](0007-native-dag-interface.md)) and task handlers in any mix. +3. **A Go bundle registers task handlers, not Dags**: `airflow.TaskHandler(dagId, taskId, fn)`, + the Go body for a task Python declares with `@task.stub`. +4. **task_id is always written out**; nothing is derived from the Go function name. +5. **Every handler takes an `airflow.Context` first**, and nothing else is injected — a struct + embedding `context.Context`, exposing `Logger()`, `Client()`, `TaskInstance()`, and `DagRun()`. +6. **Every remaining parameter is data**, bound positionally, or by field when it is a single struct: + `arg:"..."` when tagged, else the folded Go field name. +7. This **breaks** `BundleProvider`, `Registry`, and `AddDag(dagId).AddTask(fn)`, with no + deprecation alias. + +## Context + +Python owns everything but the body of a Mixed Lang task: `@task.stub` declares the task, its +arguments, and its place in the graph. The Go side has no Dag to define, so Dag vocabulary misleads, +and it occupies the `AddDag` name that [ADR 7](0007-native-dag-interface.md) needs. + +Registration is inverted today. An author declares a struct with no state, asserts it implements +`v1.BundleProvider`, fills in `RegisterDags(dagbag v1.Registry) error`, and hands the struct to +`bundlev1server.Serve` — three concepts and an empty type before a single task is declared. The two +names are also one object: `Registry` is `Bundle` plus `AddDag`, the write side of the value that +later answers task lookups at execution time. + +The shipped signature then injects `sdk.TIRunContext`, `*slog.Logger`, and clients by type in any +position, so a handler can declare no context at all and its logger arrives separately from the +context it logs against. One required context carrying the rest fixes both. Review Comment: The purpose is same as https://github.com/apache/airflow/pull/72043/changes#r4013490733 comment, update the whole context as well. ########## go-sdk/adr/0006-mixed-lang-task-handler-interface.md: ########## @@ -0,0 +1,179 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 6. Bundle registration and Mixed Lang task handlers + +Date: 2026-09-09 + +## Status + +Proposed. Keeps the argument binding shipped in #70209, replaces the provider-and-registry +registration flow around it, and reshapes the task signature, following the design review on #72043. + +## Decision + +1. **A bundle is a value the author builds.** `airflow.Bundle()` returns a `*airflow.BundleRef`; + `main` reads build, register, serve, with `bundle.Serve()` as its last statement. +2. **`bundle.Register(items ...airflow.Registration)`** is the single registration verb, taking + native Dags ([ADR 7](0007-native-dag-interface.md)) and task handlers in any mix. +3. **A Go bundle registers task handlers, not Dags**: `airflow.TaskHandler(dagId, taskId, fn)`, + the Go body for a task Python declares with `@task.stub`. +4. **task_id is always written out**; nothing is derived from the Go function name. +5. **Every handler takes an `airflow.Context` first**, and nothing else is injected — a struct + embedding `context.Context`, exposing `Logger()`, `Client()`, `TaskInstance()`, and `DagRun()`. +6. **Every remaining parameter is data**, bound positionally, or by field when it is a single struct: + `arg:"..."` when tagged, else the folded Go field name. +7. This **breaks** `BundleProvider`, `Registry`, and `AddDag(dagId).AddTask(fn)`, with no + deprecation alias. Review Comment: Yes, it's fine break, I removed the whole statement. ########## go-sdk/adr/0006-mixed-lang-task-handler-interface.md: ########## @@ -0,0 +1,179 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 6. Bundle registration and Mixed Lang task handlers + +Date: 2026-09-09 + +## Status + +Proposed. Keeps the argument binding shipped in #70209, replaces the provider-and-registry +registration flow around it, and reshapes the task signature, following the design review on #72043. + +## Decision + +1. **A bundle is a value the author builds.** `airflow.Bundle()` returns a `*airflow.BundleRef`; + `main` reads build, register, serve, with `bundle.Serve()` as its last statement. +2. **`bundle.Register(items ...airflow.Registration)`** is the single registration verb, taking + native Dags ([ADR 7](0007-native-dag-interface.md)) and task handlers in any mix. +3. **A Go bundle registers task handlers, not Dags**: `airflow.TaskHandler(dagId, taskId, fn)`, + the Go body for a task Python declares with `@task.stub`. +4. **task_id is always written out**; nothing is derived from the Go function name. +5. **Every handler takes an `airflow.Context` first**, and nothing else is injected — a struct Review Comment: This part is AI slop that I didn't catch, the purpose of having `airflow.Context` is user could invoke client or retrieving the TaskInstance without invoking the client with `context.Context` again. ########## go-sdk/adr/0006-mixed-lang-task-handler-interface.md: ########## @@ -0,0 +1,179 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 6. Bundle registration and Mixed Lang task handlers + +Date: 2026-09-09 + +## Status + +Proposed. Keeps the argument binding shipped in #70209, replaces the provider-and-registry +registration flow around it, and reshapes the task signature, following the design review on #72043. + +## Decision + +1. **A bundle is a value the author builds.** `airflow.Bundle()` returns a `*airflow.BundleRef`; + `main` reads build, register, serve, with `bundle.Serve()` as its last statement. +2. **`bundle.Register(items ...airflow.Registration)`** is the single registration verb, taking + native Dags ([ADR 7](0007-native-dag-interface.md)) and task handlers in any mix. +3. **A Go bundle registers task handlers, not Dags**: `airflow.TaskHandler(dagId, taskId, fn)`, + the Go body for a task Python declares with `@task.stub`. +4. **task_id is always written out**; nothing is derived from the Go function name. Review Comment: I kept the context of why `airflow.TaskHandler(dagId, taskId, fn)` is a better shape in the ADR now. ########## go-sdk/adr/0006-mixed-lang-task-handler-interface.md: ########## @@ -0,0 +1,179 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 6. Bundle registration and Mixed Lang task handlers + +Date: 2026-09-09 + +## Status + +Proposed. Keeps the argument binding shipped in #70209, replaces the provider-and-registry +registration flow around it, and reshapes the task signature, following the design review on #72043. + +## Decision + +1. **A bundle is a value the author builds.** `airflow.Bundle()` returns a `*airflow.BundleRef`; + `main` reads build, register, serve, with `bundle.Serve()` as its last statement. +2. **`bundle.Register(items ...airflow.Registration)`** is the single registration verb, taking + native Dags ([ADR 7](0007-native-dag-interface.md)) and task handlers in any mix. +3. **A Go bundle registers task handlers, not Dags**: `airflow.TaskHandler(dagId, taskId, fn)`, + the Go body for a task Python declares with `@task.stub`. +4. **task_id is always written out**; nothing is derived from the Go function name. +5. **Every handler takes an `airflow.Context` first**, and nothing else is injected — a struct + embedding `context.Context`, exposing `Logger()`, `Client()`, `TaskInstance()`, and `DagRun()`. +6. **Every remaining parameter is data**, bound positionally, or by field when it is a single struct: + `arg:"..."` when tagged, else the folded Go field name. +7. This **breaks** `BundleProvider`, `Registry`, and `AddDag(dagId).AddTask(fn)`, with no + deprecation alias. + +## Context + +Python owns everything but the body of a Mixed Lang task: `@task.stub` declares the task, its +arguments, and its place in the graph. The Go side has no Dag to define, so Dag vocabulary misleads, +and it occupies the `AddDag` name that [ADR 7](0007-native-dag-interface.md) needs. + +Registration is inverted today. An author declares a struct with no state, asserts it implements +`v1.BundleProvider`, fills in `RegisterDags(dagbag v1.Registry) error`, and hands the struct to +`bundlev1server.Serve` — three concepts and an empty type before a single task is declared. The two +names are also one object: `Registry` is `Bundle` plus `AddDag`, the write side of the value that +later answers task lookups at execution time. + +The shipped signature then injects `sdk.TIRunContext`, `*slog.Logger`, and clients by type in any +position, so a handler can declare no context at all and its logger arrives separately from the +context it logs against. One required context carrying the rest fixes both. + +## Example + +```go +func main() { + bundle := airflow.Bundle() + + bundle.Register( + nativeEtl, // *airflow.DagRef, from ADR 7 Review Comment: Removed all the ADR7 references. ########## go-sdk/adr/0006-mixed-lang-task-handler-interface.md: ########## @@ -0,0 +1,179 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 6. Bundle registration and Mixed Lang task handlers + +Date: 2026-09-09 + +## Status + +Proposed. Keeps the argument binding shipped in #70209, replaces the provider-and-registry +registration flow around it, and reshapes the task signature, following the design review on #72043. + +## Decision + +1. **A bundle is a value the author builds.** `airflow.Bundle()` returns a `*airflow.BundleRef`; + `main` reads build, register, serve, with `bundle.Serve()` as its last statement. +2. **`bundle.Register(items ...airflow.Registration)`** is the single registration verb, taking + native Dags ([ADR 7](0007-native-dag-interface.md)) and task handlers in any mix. +3. **A Go bundle registers task handlers, not Dags**: `airflow.TaskHandler(dagId, taskId, fn)`, + the Go body for a task Python declares with `@task.stub`. +4. **task_id is always written out**; nothing is derived from the Go function name. +5. **Every handler takes an `airflow.Context` first**, and nothing else is injected — a struct + embedding `context.Context`, exposing `Logger()`, `Client()`, `TaskInstance()`, and `DagRun()`. +6. **Every remaining parameter is data**, bound positionally, or by field when it is a single struct: + `arg:"..."` when tagged, else the folded Go field name. +7. This **breaks** `BundleProvider`, `Registry`, and `AddDag(dagId).AddTask(fn)`, with no + deprecation alias. + +## Context + +Python owns everything but the body of a Mixed Lang task: `@task.stub` declares the task, its +arguments, and its place in the graph. The Go side has no Dag to define, so Dag vocabulary misleads, +and it occupies the `AddDag` name that [ADR 7](0007-native-dag-interface.md) needs. + +Registration is inverted today. An author declares a struct with no state, asserts it implements +`v1.BundleProvider`, fills in `RegisterDags(dagbag v1.Registry) error`, and hands the struct to +`bundlev1server.Serve` — three concepts and an empty type before a single task is declared. The two +names are also one object: `Registry` is `Bundle` plus `AddDag`, the write side of the value that +later answers task lookups at execution time. + +The shipped signature then injects `sdk.TIRunContext`, `*slog.Logger`, and clients by type in any +position, so a handler can declare no context at all and its logger arrives separately from the +context it logs against. One required context carrying the rest fixes both. + +## Example + +```go +func main() { + bundle := airflow.Bundle() + + bundle.Register( + nativeEtl, // *airflow.DagRef, from ADR 7 + airflow.TaskHandler("py_etl", "transform", transform), + airflow.TaskHandler("py_etl", "via_struct_arg_tag", ViaStructArgTag), + ) + + if err := bundle.Serve(); err != nil { + log.Fatal(err) + } +} +``` + +Registration can be spread across packages, either by passing the bundle along or by returning +`[]airflow.Registration` for the caller to splat: `bundle.Register(taskflowbinding.Handlers()...)`. + +Three ways a Go function receives a stub task's data, all live in `go-sdk/example/bundle/`. + +**Flat positional**, for `def transform(country: str, extracted: dict)`: + +```go +func transform(actx airflow.Context, country string, extracted map[string]any) error { + actx.Logger().Info("transforming", "country", country, "try", actx.TaskInstance().TryNumber) + + threshold, err := actx.Client().GetVariable(actx, "etl_threshold") + if err != nil { + return err + } + return writeRows(actx, extracted, threshold) +} +``` + +**A single `arg:`-tagged struct**, for `def via_struct_arg_tag(region_code: str, threshold: float)`: + +```go +type ViaStructArgTagInput struct { + Region string `arg:"region_code"` + Threshold float64 `arg:"threshold"` +} + +func ViaStructArgTag(actx airflow.Context, input ViaStructArgTagInput) (any, error) +``` + +**A single untagged struct**, where the field name folds to the Python argument: `RegionCode` +lowercased with underscores stripped is `regioncode`, which matches `region_code`. + +## Consequences + +- `BundleProvider`, the empty provider struct, and the `bundlev1server` package all disappear from an + author's `main`; `Serve` becomes a method on the value they already hold. +- **Registration closes when `Serve` is called.** Registering afterwards is a programming error and + panics, like every other registration-time check in these ADRs. +- Every existing call site changes, acceptable only because the SDK's README already warns its APIs + "may change between releases without notice." +- Requiring `airflow.Context` turns a class of mistake into a compile error: a test calling + `myTask(context.Background(), ...)` no longer builds, where a package-level `airflow.Logger(ctx)` + accessor over a plain context would have compiled and then failed at run time on a missing value. Review Comment: Moving this to the rejected alternative part, since this is one of the `airflow.Logger(ctx)` approach that we discussed but it got rejected. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. + +## Decision + +1. **One Dag type, constructed then registered.** `airflow.Dag(spec)` returns a `*airflow.DagRef` + that is complete before `bundle.Register(dag)` takes it — the same verb that registers Mixed Lang + task handlers ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Naming rule: + `airflow.X(...)` constructs, `*airflow.XRef` is the handle. +2. **Tasks register through `dag.Task(fn any, opts ...airflow.TaskOption)`**, returning a + `*airflow.TaskRef`. `airflow.Inputs(...)` and a bare `airflow.TaskSpec{}` both satisfy + `TaskOption`, which is how one variadic carries both. +3. **`airflow.Inputs(refs...)` is data and an edge at once**, binding positionally after the context. Review Comment: Rephrase the statement as "`airflow.Inputs(refs...)` declares the data and the edge in one call", it should be more clear now. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. + +## Decision + +1. **One Dag type, constructed then registered.** `airflow.Dag(spec)` returns a `*airflow.DagRef` + that is complete before `bundle.Register(dag)` takes it — the same verb that registers Mixed Lang + task handlers ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Naming rule: + `airflow.X(...)` constructs, `*airflow.XRef` is the handle. +2. **Tasks register through `dag.Task(fn any, opts ...airflow.TaskOption)`**, returning a + `*airflow.TaskRef`. `airflow.Inputs(...)` and a bare `airflow.TaskSpec{}` both satisfy + `TaskOption`, which is how one variadic carries both. +3. **`airflow.Inputs(refs...)` is data and an edge at once**, binding positionally after the context. +4. **`Before` and `After` are order-only edges on `airflow.Node`**, which both `*airflow.TaskRef` + and `*airflow.TaskGroupRef` satisfy, so a task and a whole group are equally an edge endpoint. + They are the Go pair for `>>` and `<<`, both variadic so a single call fans out. +5. **Trigger rules belong to the task**, as `airflow.TaskSpec{TriggerRule: ...}`, never to an edge. +6. **Everything an author writes comes from one `airflow` package.** +7. **No Go-native deferral**, and none is needed: the constructs that defer are DSL tasks Python + executes. + +## Context + +A native Dag is authored entirely in Go — schedule, tasks, and dependencies — and serializes into the +Dag JSON a Python Dag would produce. There is one Dag type, as in Python; the Mixed Lang case +registers task handlers instead ([ADR 6](0006-mixed-lang-task-handler-interface.md)) because it +defines no Dag. Dependencies between Go functions have to be typed rather than looked up by task ID, +and a Dag should read like Go rather than transliterated Python. The proposed interface spread its +surface across `v1`, `sdk`, and `slog`, published a half-built Dag to the registry and mutated it Review Comment: Added all the public interface signatures. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. + +## Decision + +1. **One Dag type, constructed then registered.** `airflow.Dag(spec)` returns a `*airflow.DagRef` + that is complete before `bundle.Register(dag)` takes it — the same verb that registers Mixed Lang + task handlers ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Naming rule: + `airflow.X(...)` constructs, `*airflow.XRef` is the handle. +2. **Tasks register through `dag.Task(fn any, opts ...airflow.TaskOption)`**, returning a + `*airflow.TaskRef`. `airflow.Inputs(...)` and a bare `airflow.TaskSpec{}` both satisfy + `TaskOption`, which is how one variadic carries both. +3. **`airflow.Inputs(refs...)` is data and an edge at once**, binding positionally after the context. +4. **`Before` and `After` are order-only edges on `airflow.Node`**, which both `*airflow.TaskRef` + and `*airflow.TaskGroupRef` satisfy, so a task and a whole group are equally an edge endpoint. + They are the Go pair for `>>` and `<<`, both variadic so a single call fans out. +5. **Trigger rules belong to the task**, as `airflow.TaskSpec{TriggerRule: ...}`, never to an edge. Review Comment: Yes, added the signature to show the relationship of TaskSpec and TaskOption. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. + +## Decision + +1. **One Dag type, constructed then registered.** `airflow.Dag(spec)` returns a `*airflow.DagRef` + that is complete before `bundle.Register(dag)` takes it — the same verb that registers Mixed Lang + task handlers ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Naming rule: + `airflow.X(...)` constructs, `*airflow.XRef` is the handle. +2. **Tasks register through `dag.Task(fn any, opts ...airflow.TaskOption)`**, returning a + `*airflow.TaskRef`. `airflow.Inputs(...)` and a bare `airflow.TaskSpec{}` both satisfy + `TaskOption`, which is how one variadic carries both. +3. **`airflow.Inputs(refs...)` is data and an edge at once**, binding positionally after the context. +4. **`Before` and `After` are order-only edges on `airflow.Node`**, which both `*airflow.TaskRef` + and `*airflow.TaskGroupRef` satisfy, so a task and a whole group are equally an edge endpoint. + They are the Go pair for `>>` and `<<`, both variadic so a single call fans out. +5. **Trigger rules belong to the task**, as `airflow.TaskSpec{TriggerRule: ...}`, never to an edge. +6. **Everything an author writes comes from one `airflow` package.** +7. **No Go-native deferral**, and none is needed: the constructs that defer are DSL tasks Python + executes. + +## Context + +A native Dag is authored entirely in Go — schedule, tasks, and dependencies — and serializes into the +Dag JSON a Python Dag would produce. There is one Dag type, as in Python; the Mixed Lang case +registers task handlers instead ([ADR 6](0006-mixed-lang-task-handler-interface.md)) because it +defines no Dag. Dependencies between Go functions have to be typed rather than looked up by task ID, +and a Dag should read like Go rather than transliterated Python. The proposed interface spread its +surface across `v1`, `sdk`, and `slog`, published a half-built Dag to the registry and mutated it +afterwards, and could declare an edge in only one direction. + +## Example + +Both forms build the same graph; which one an author writes depends on whether the edge carries a +value. + +**Data dependencies — the TaskFlow equivalent.** `airflow.Inputs` passes an upstream's return value +in and declares the edge in one call, as calling one TaskFlow function with another's output does in +Python (`extracted = extract(); transformed = transform(extracted); load(transformed)`). + +```go +dag := airflow.Dag(airflow.DagSpec{DagId: "etl", Schedule: "@daily"}) + +extracted := dag.Task(extract) +transformed := dag.Task(transform, airflow.Inputs(extracted)) +dag.Task(load, airflow.Inputs(transformed), airflow.TaskSpec{Retries: 2}) Review Comment: Added the signature of Inputs. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. + +## Decision + +1. **One Dag type, constructed then registered.** `airflow.Dag(spec)` returns a `*airflow.DagRef` + that is complete before `bundle.Register(dag)` takes it — the same verb that registers Mixed Lang + task handlers ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Naming rule: + `airflow.X(...)` constructs, `*airflow.XRef` is the handle. +2. **Tasks register through `dag.Task(fn any, opts ...airflow.TaskOption)`**, returning a + `*airflow.TaskRef`. `airflow.Inputs(...)` and a bare `airflow.TaskSpec{}` both satisfy + `TaskOption`, which is how one variadic carries both. +3. **`airflow.Inputs(refs...)` is data and an edge at once**, binding positionally after the context. +4. **`Before` and `After` are order-only edges on `airflow.Node`**, which both `*airflow.TaskRef` + and `*airflow.TaskGroupRef` satisfy, so a task and a whole group are equally an edge endpoint. + They are the Go pair for `>>` and `<<`, both variadic so a single call fans out. +5. **Trigger rules belong to the task**, as `airflow.TaskSpec{TriggerRule: ...}`, never to an edge. +6. **Everything an author writes comes from one `airflow` package.** +7. **No Go-native deferral**, and none is needed: the constructs that defer are DSL tasks Python + executes. + +## Context + +A native Dag is authored entirely in Go — schedule, tasks, and dependencies — and serializes into the +Dag JSON a Python Dag would produce. There is one Dag type, as in Python; the Mixed Lang case +registers task handlers instead ([ADR 6](0006-mixed-lang-task-handler-interface.md)) because it +defines no Dag. Dependencies between Go functions have to be typed rather than looked up by task ID, +and a Dag should read like Go rather than transliterated Python. The proposed interface spread its +surface across `v1`, `sdk`, and `slog`, published a half-built Dag to the registry and mutated it +afterwards, and could declare an edge in only one direction. + +## Example + +Both forms build the same graph; which one an author writes depends on whether the edge carries a +value. + +**Data dependencies — the TaskFlow equivalent.** `airflow.Inputs` passes an upstream's return value +in and declares the edge in one call, as calling one TaskFlow function with another's output does in +Python (`extracted = extract(); transformed = transform(extracted); load(transformed)`). + +```go +dag := airflow.Dag(airflow.DagSpec{DagId: "etl", Schedule: "@daily"}) + +extracted := dag.Task(extract) +transformed := dag.Task(transform, airflow.Inputs(extracted)) +dag.Task(load, airflow.Inputs(transformed), airflow.TaskSpec{Retries: 2}) + +bundle.Register(dag) +``` + +```go +func extract(actx airflow.Context) (Result, error) { + return Result{Message: "native Dag data"}, nil +} Review Comment: It's a user defined struct, added the definition of `Result` in the code example as well. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. + +## Decision + +1. **One Dag type, constructed then registered.** `airflow.Dag(spec)` returns a `*airflow.DagRef` + that is complete before `bundle.Register(dag)` takes it — the same verb that registers Mixed Lang + task handlers ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Naming rule: + `airflow.X(...)` constructs, `*airflow.XRef` is the handle. +2. **Tasks register through `dag.Task(fn any, opts ...airflow.TaskOption)`**, returning a + `*airflow.TaskRef`. `airflow.Inputs(...)` and a bare `airflow.TaskSpec{}` both satisfy + `TaskOption`, which is how one variadic carries both. +3. **`airflow.Inputs(refs...)` is data and an edge at once**, binding positionally after the context. +4. **`Before` and `After` are order-only edges on `airflow.Node`**, which both `*airflow.TaskRef` + and `*airflow.TaskGroupRef` satisfy, so a task and a whole group are equally an edge endpoint. + They are the Go pair for `>>` and `<<`, both variadic so a single call fans out. +5. **Trigger rules belong to the task**, as `airflow.TaskSpec{TriggerRule: ...}`, never to an edge. +6. **Everything an author writes comes from one `airflow` package.** +7. **No Go-native deferral**, and none is needed: the constructs that defer are DSL tasks Python + executes. + +## Context + +A native Dag is authored entirely in Go — schedule, tasks, and dependencies — and serializes into the +Dag JSON a Python Dag would produce. There is one Dag type, as in Python; the Mixed Lang case +registers task handlers instead ([ADR 6](0006-mixed-lang-task-handler-interface.md)) because it +defines no Dag. Dependencies between Go functions have to be typed rather than looked up by task ID, +and a Dag should read like Go rather than transliterated Python. The proposed interface spread its +surface across `v1`, `sdk`, and `slog`, published a half-built Dag to the registry and mutated it +afterwards, and could declare an edge in only one direction. + +## Example + +Both forms build the same graph; which one an author writes depends on whether the edge carries a +value. + +**Data dependencies — the TaskFlow equivalent.** `airflow.Inputs` passes an upstream's return value +in and declares the edge in one call, as calling one TaskFlow function with another's output does in +Python (`extracted = extract(); transformed = transform(extracted); load(transformed)`). + +```go +dag := airflow.Dag(airflow.DagSpec{DagId: "etl", Schedule: "@daily"}) + +extracted := dag.Task(extract) +transformed := dag.Task(transform, airflow.Inputs(extracted)) +dag.Task(load, airflow.Inputs(transformed), airflow.TaskSpec{Retries: 2}) + +bundle.Register(dag) +``` + +```go +func extract(actx airflow.Context) (Result, error) { + return Result{Message: "native Dag data"}, nil +} + +func transform(actx airflow.Context, extracted Result) (Result, error) { + return Result{Message: "transformed " + extracted.Message}, nil +} + +func load(actx airflow.Context, transformed Result) error { return nil } +``` + +**Order-only dependencies — the `>>` and `<<` equivalent.** For tasks that must be ordered but +exchange no data; the functions take no parameter for such an edge. + +```go +loaded := dag.Task(load, airflow.Inputs(transformed)) +cleaned := dag.Task(cleanup, airflow.TaskSpec{TriggerRule: airflow.AllDone}) +staging := dag.TaskGroup("staging") // a group carries edges like a task + +loaded.Before(dag.Task(notify), cleaned) // loaded >> [notify, cleanup] +cleaned.After(extracted) // cleanup << extracted +staging.Before(loaded) // staging >> load +``` + +## Consequences + +- **A cycle check becomes necessary.** `Inputs` alone cannot express one, since a `*TaskRef` exists + only after its own `dag.Task(...)` returns. `Before`/`After` link two existing refs in either + direction, so `b := dag.Task(B, airflow.Inputs(a)); b.Before(a)` is a genuine cycle in accepted + syntax, and registration has to reject it. The check has to see through groups, since a group edge + stands for edges into and out of every task the group holds. +- **A count or type mismatch panics at registration**, not at run time, because each `*TaskRef` + carries its recorded output type. +- **A chained `Before`/`After` expression is an `airflow.Node`, not a `*TaskRef`.** Go has no + covariant returns, so a method declared on the interface returns the interface. Take the ref from + `dag.Task(...)` when it is needed for `Inputs`; the style above calls both verbs as statements and + never reads the result. +- **Cancellation works even though deferral does not**: `actx.Done()` fires on supervisor shutdown + ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Concurrency primitives stay out of scope + until a Go-native task itself needs to wait. Review Comment: It's an AI slop I didn't catch, I remove the statement. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. + +## Decision + +1. **One Dag type, constructed then registered.** `airflow.Dag(spec)` returns a `*airflow.DagRef` + that is complete before `bundle.Register(dag)` takes it — the same verb that registers Mixed Lang + task handlers ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Naming rule: + `airflow.X(...)` constructs, `*airflow.XRef` is the handle. +2. **Tasks register through `dag.Task(fn any, opts ...airflow.TaskOption)`**, returning a + `*airflow.TaskRef`. `airflow.Inputs(...)` and a bare `airflow.TaskSpec{}` both satisfy + `TaskOption`, which is how one variadic carries both. +3. **`airflow.Inputs(refs...)` is data and an edge at once**, binding positionally after the context. +4. **`Before` and `After` are order-only edges on `airflow.Node`**, which both `*airflow.TaskRef` + and `*airflow.TaskGroupRef` satisfy, so a task and a whole group are equally an edge endpoint. + They are the Go pair for `>>` and `<<`, both variadic so a single call fans out. +5. **Trigger rules belong to the task**, as `airflow.TaskSpec{TriggerRule: ...}`, never to an edge. +6. **Everything an author writes comes from one `airflow` package.** +7. **No Go-native deferral**, and none is needed: the constructs that defer are DSL tasks Python + executes. + +## Context + +A native Dag is authored entirely in Go — schedule, tasks, and dependencies — and serializes into the +Dag JSON a Python Dag would produce. There is one Dag type, as in Python; the Mixed Lang case +registers task handlers instead ([ADR 6](0006-mixed-lang-task-handler-interface.md)) because it +defines no Dag. Dependencies between Go functions have to be typed rather than looked up by task ID, +and a Dag should read like Go rather than transliterated Python. The proposed interface spread its +surface across `v1`, `sdk`, and `slog`, published a half-built Dag to the registry and mutated it +afterwards, and could declare an edge in only one direction. + +## Example + +Both forms build the same graph; which one an author writes depends on whether the edge carries a +value. + +**Data dependencies — the TaskFlow equivalent.** `airflow.Inputs` passes an upstream's return value +in and declares the edge in one call, as calling one TaskFlow function with another's output does in +Python (`extracted = extract(); transformed = transform(extracted); load(transformed)`). + +```go +dag := airflow.Dag(airflow.DagSpec{DagId: "etl", Schedule: "@daily"}) + +extracted := dag.Task(extract) +transformed := dag.Task(transform, airflow.Inputs(extracted)) +dag.Task(load, airflow.Inputs(transformed), airflow.TaskSpec{Retries: 2}) + +bundle.Register(dag) +``` + +```go +func extract(actx airflow.Context) (Result, error) { + return Result{Message: "native Dag data"}, nil +} + +func transform(actx airflow.Context, extracted Result) (Result, error) { + return Result{Message: "transformed " + extracted.Message}, nil +} + +func load(actx airflow.Context, transformed Result) error { return nil } +``` + +**Order-only dependencies — the `>>` and `<<` equivalent.** For tasks that must be ordered but +exchange no data; the functions take no parameter for such an edge. + +```go +loaded := dag.Task(load, airflow.Inputs(transformed)) +cleaned := dag.Task(cleanup, airflow.TaskSpec{TriggerRule: airflow.AllDone}) Review Comment: Agreed, recorded down in the ADR. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. Review Comment: Only keep all the status as proposed. ########## go-sdk/adr/0006-mixed-lang-task-handler-interface.md: ########## @@ -0,0 +1,179 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 6. Bundle registration and Mixed Lang task handlers + +Date: 2026-09-09 + +## Status + +Proposed. Keeps the argument binding shipped in #70209, replaces the provider-and-registry +registration flow around it, and reshapes the task signature, following the design review on #72043. + +## Decision + +1. **A bundle is a value the author builds.** `airflow.Bundle()` returns a `*airflow.BundleRef`; + `main` reads build, register, serve, with `bundle.Serve()` as its last statement. +2. **`bundle.Register(items ...airflow.Registration)`** is the single registration verb, taking + native Dags ([ADR 7](0007-native-dag-interface.md)) and task handlers in any mix. +3. **A Go bundle registers task handlers, not Dags**: `airflow.TaskHandler(dagId, taskId, fn)`, + the Go body for a task Python declares with `@task.stub`. +4. **task_id is always written out**; nothing is derived from the Go function name. +5. **Every handler takes an `airflow.Context` first**, and nothing else is injected — a struct + embedding `context.Context`, exposing `Logger()`, `Client()`, `TaskInstance()`, and `DagRun()`. +6. **Every remaining parameter is data**, bound positionally, or by field when it is a single struct: + `arg:"..."` when tagged, else the folded Go field name. +7. This **breaks** `BundleProvider`, `Registry`, and `AddDag(dagId).AddTask(fn)`, with no + deprecation alias. + +## Context + +Python owns everything but the body of a Mixed Lang task: `@task.stub` declares the task, its +arguments, and its place in the graph. The Go side has no Dag to define, so Dag vocabulary misleads, +and it occupies the `AddDag` name that [ADR 7](0007-native-dag-interface.md) needs. + +Registration is inverted today. An author declares a struct with no state, asserts it implements +`v1.BundleProvider`, fills in `RegisterDags(dagbag v1.Registry) error`, and hands the struct to +`bundlev1server.Serve` — three concepts and an empty type before a single task is declared. The two +names are also one object: `Registry` is `Bundle` plus `AddDag`, the write side of the value that +later answers task lookups at execution time. + +The shipped signature then injects `sdk.TIRunContext`, `*slog.Logger`, and clients by type in any +position, so a handler can declare no context at all and its logger arrives separately from the +context it logs against. One required context carrying the rest fixes both. + +## Example + +```go +func main() { + bundle := airflow.Bundle() + + bundle.Register( + nativeEtl, // *airflow.DagRef, from ADR 7 + airflow.TaskHandler("py_etl", "transform", transform), + airflow.TaskHandler("py_etl", "via_struct_arg_tag", ViaStructArgTag), + ) + + if err := bundle.Serve(); err != nil { + log.Fatal(err) + } +} +``` + +Registration can be spread across packages, either by passing the bundle along or by returning +`[]airflow.Registration` for the caller to splat: `bundle.Register(taskflowbinding.Handlers()...)`. + +Three ways a Go function receives a stub task's data, all live in `go-sdk/example/bundle/`. + +**Flat positional**, for `def transform(country: str, extracted: dict)`: + +```go +func transform(actx airflow.Context, country string, extracted map[string]any) error { + actx.Logger().Info("transforming", "country", country, "try", actx.TaskInstance().TryNumber) + + threshold, err := actx.Client().GetVariable(actx, "etl_threshold") + if err != nil { + return err + } + return writeRows(actx, extracted, threshold) +} +``` + +**A single `arg:`-tagged struct**, for `def via_struct_arg_tag(region_code: str, threshold: float)`: + +```go +type ViaStructArgTagInput struct { + Region string `arg:"region_code"` + Threshold float64 `arg:"threshold"` +} + +func ViaStructArgTag(actx airflow.Context, input ViaStructArgTagInput) (any, error) +``` + +**A single untagged struct**, where the field name folds to the Python argument: `RegionCode` +lowercased with underscores stripped is `regioncode`, which matches `region_code`. + +## Consequences + +- `BundleProvider`, the empty provider struct, and the `bundlev1server` package all disappear from an + author's `main`; `Serve` becomes a method on the value they already hold. +- **Registration closes when `Serve` is called.** Registering afterwards is a programming error and + panics, like every other registration-time check in these ADRs. +- Every existing call site changes, acceptable only because the SDK's README already warns its APIs + "may change between releases without notice." +- Requiring `airflow.Context` turns a class of mistake into a compile error: a test calling + `myTask(context.Background(), ...)` no longer builds, where a package-level `airflow.Logger(ctx)` + accessor over a plain context would have compiled and then failed at run time on a missing value. +- Graceful termination needs no unwrapping — `actx.Done()` fires on supervisor shutdown, and + `http.NewRequestWithContext(actx, ...)` accepts it — while cleanup that must outlive cancellation + uses `context.WithoutCancel(actx)`. +- A helper typed as a plain `context.Context` recovers the methods with + `airflow.FromContext(ctx) (airflow.Context, bool)`. + +## Appendix: Implementation Notes + +- **One verb, over a sealed interface.** `Bundle.Register(items ...airflow.Registration)` accepts + both kinds because `*airflow.DagRef` and `airflow.TaskHandlerRef` satisfy `Registration`, an + exported interface whose only method is unexported. That closes the set to the SDK's own types, and + an author never writes the name — the same technique as `airflow.TaskOption` in + [ADR 7](0007-native-dag-interface.md). Two verbs would have split what a bundle provides across + separate calls for no gain in safety, since the interface already rejects anything else at compile + time. Variadic, rather than one handler per call, which repeats the dag_id per task. +- **`airflow.Bundle()` follows the same rule as `Dag` and `Task`**: the constructor takes the noun, + the handle is `*airflow.BundleRef`. Carving out an exception — `NewBundle()` returning + `*airflow.Bundle` — would buy a better type name for helper signatures, but the recommended way to + spread registration is a package returning `[]airflow.Registration` rather than passing the bundle + around, so the type name rarely appears. Keeping the rule mechanical also leaves room for a + `BundleSpec` argument later, exactly as `airflow.Dag(spec)` takes one. It takes none today: the + bundle's name is Airflow-side configuration, arriving as `bundle_name` on the wire, and the + manifest the executable prints carries only the SDK version and the Dag list + (`go-sdk/internal/airflowmetadata/airflowmetadata.go`). +- **`Serve` keeps both modes the executable already has**, selected by flags in + `go-sdk/bundle/bundlev1/bundlev1server/server.go`: `--airflow-metadata` prints the manifest JSON + ([ADR 2](0002-use-go-tool-directive-for-bundle-packer.md), + [ADR 4](0004-self-contained-executable-bundle.md)), and `--comm`/`--logs` runs the + msgpack-over-IPC coordinator path ([ADR 3](0003-coordinator-protocol-msgpack-ipc.md)). It returns + an error rather than exiting, so `main` reports it. +- **The value-first shape already half exists**: `bundlev1.New() Registry` + (`go-sdk/bundle/bundlev1/registry.go`) builds a registry outside the provider callback, documented + as useful for unit-testing a `RegisterDags` implementation. Making it the only shape removes the + callback rather than adding a mechanism. +- **A struct embedding `context.Context`, not an interface.** The context package's advice against + [storing a Context in a struct](https://pkg.go.dev/context#hdr-Contexts_and_structs) is about + domain types — a `DagRun` holding a request-scoped context — not about a purpose-built context + type; `context.WithValue` itself returns a struct. A struct fits because only the SDK implements + this type: methods can be added as the context grows without breaking anyone, and godoc has a + concrete type to document. The shipped `sdk.TIRunContext` (`go-sdk/sdk/context.go`) is an + interface, justified in its doc comment by that same misreading, so the comment is corrected along + with the change. +- **The constructor is the supported way to build a context.** Every field but the embedded context + is unexported, so no caller can substitute a logger or client; what is worth faking in a task test + is already an interface (`sdk.Client`, and the logger through an `slog.Handler`). Embedding does + leave `Context` exported as a field name, so `airflow.Context{Context: ctx}` compiles and yields a + value whose `Logger()` and `Client()` are nil; closing that off would mean an unexported field plus + four delegating methods in place of promotion. +- **Binding** lives in `go-sdk/pkg/binding/binding.go`: `classifyParam` for the signature, the field + fallback as `strings.ToLower(strings.ReplaceAll(name, "_", ""))`. A struct carrying `arg:` tags + cannot be mixed with other data parameters and is rejected at registration; an untagged struct has + no such guard and is decoded positionally as one value. +- **Cancellation and the value lookup already exist.** `pkg/execution/server.go` traps + `SIGINT`/`SIGTERM` into the context the runtime binds, and `pkg/execution/task_runner.go` stores + the client and run context as values on it — which is what makes `airflow.FromContext` a typed + lookup rather than new plumbing. A task ignoring `actx.Done()` is still stopped by the supervisor's + follow-up `SIGKILL`. Review Comment: Removed all the side notes from the ADRs. ########## go-sdk/adr/0006-mixed-lang-task-handler-interface.md: ########## @@ -0,0 +1,179 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 6. Bundle registration and Mixed Lang task handlers + +Date: 2026-09-09 + +## Status + +Proposed. Keeps the argument binding shipped in #70209, replaces the provider-and-registry +registration flow around it, and reshapes the task signature, following the design review on #72043. + +## Decision + +1. **A bundle is a value the author builds.** `airflow.Bundle()` returns a `*airflow.BundleRef`; + `main` reads build, register, serve, with `bundle.Serve()` as its last statement. +2. **`bundle.Register(items ...airflow.Registration)`** is the single registration verb, taking + native Dags ([ADR 7](0007-native-dag-interface.md)) and task handlers in any mix. +3. **A Go bundle registers task handlers, not Dags**: `airflow.TaskHandler(dagId, taskId, fn)`, + the Go body for a task Python declares with `@task.stub`. +4. **task_id is always written out**; nothing is derived from the Go function name. +5. **Every handler takes an `airflow.Context` first**, and nothing else is injected — a struct + embedding `context.Context`, exposing `Logger()`, `Client()`, `TaskInstance()`, and `DagRun()`. +6. **Every remaining parameter is data**, bound positionally, or by field when it is a single struct: + `arg:"..."` when tagged, else the folded Go field name. +7. This **breaks** `BundleProvider`, `Registry`, and `AddDag(dagId).AddTask(fn)`, with no + deprecation alias. + +## Context + +Python owns everything but the body of a Mixed Lang task: `@task.stub` declares the task, its +arguments, and its place in the graph. The Go side has no Dag to define, so Dag vocabulary misleads, +and it occupies the `AddDag` name that [ADR 7](0007-native-dag-interface.md) needs. + +Registration is inverted today. An author declares a struct with no state, asserts it implements +`v1.BundleProvider`, fills in `RegisterDags(dagbag v1.Registry) error`, and hands the struct to +`bundlev1server.Serve` — three concepts and an empty type before a single task is declared. The two +names are also one object: `Registry` is `Bundle` plus `AddDag`, the write side of the value that +later answers task lookups at execution time. + +The shipped signature then injects `sdk.TIRunContext`, `*slog.Logger`, and clients by type in any +position, so a handler can declare no context at all and its logger arrives separately from the +context it logs against. One required context carrying the rest fixes both. + +## Example + +```go +func main() { + bundle := airflow.Bundle() + + bundle.Register( + nativeEtl, // *airflow.DagRef, from ADR 7 + airflow.TaskHandler("py_etl", "transform", transform), + airflow.TaskHandler("py_etl", "via_struct_arg_tag", ViaStructArgTag), + ) + + if err := bundle.Serve(); err != nil { + log.Fatal(err) + } +} +``` + +Registration can be spread across packages, either by passing the bundle along or by returning +`[]airflow.Registration` for the caller to splat: `bundle.Register(taskflowbinding.Handlers()...)`. + +Three ways a Go function receives a stub task's data, all live in `go-sdk/example/bundle/`. + +**Flat positional**, for `def transform(country: str, extracted: dict)`: + +```go +func transform(actx airflow.Context, country string, extracted map[string]any) error { + actx.Logger().Info("transforming", "country", country, "try", actx.TaskInstance().TryNumber) + + threshold, err := actx.Client().GetVariable(actx, "etl_threshold") + if err != nil { + return err + } + return writeRows(actx, extracted, threshold) +} +``` + +**A single `arg:`-tagged struct**, for `def via_struct_arg_tag(region_code: str, threshold: float)`: + +```go +type ViaStructArgTagInput struct { + Region string `arg:"region_code"` + Threshold float64 `arg:"threshold"` +} + +func ViaStructArgTag(actx airflow.Context, input ViaStructArgTagInput) (any, error) +``` + +**A single untagged struct**, where the field name folds to the Python argument: `RegionCode` +lowercased with underscores stripped is `regioncode`, which matches `region_code`. + +## Consequences + +- `BundleProvider`, the empty provider struct, and the `bundlev1server` package all disappear from an + author's `main`; `Serve` becomes a method on the value they already hold. Review Comment: Rephrased as "Replace `BundleProvider`and`Registry` with single `Bundle` interface" in order to reduce the new terms for users. ########## go-sdk/adr/0006-mixed-lang-task-handler-interface.md: ########## @@ -0,0 +1,179 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 6. Bundle registration and Mixed Lang task handlers + +Date: 2026-09-09 + +## Status + +Proposed. Keeps the argument binding shipped in #70209, replaces the provider-and-registry +registration flow around it, and reshapes the task signature, following the design review on #72043. + +## Decision + +1. **A bundle is a value the author builds.** `airflow.Bundle()` returns a `*airflow.BundleRef`; + `main` reads build, register, serve, with `bundle.Serve()` as its last statement. +2. **`bundle.Register(items ...airflow.Registration)`** is the single registration verb, taking + native Dags ([ADR 7](0007-native-dag-interface.md)) and task handlers in any mix. +3. **A Go bundle registers task handlers, not Dags**: `airflow.TaskHandler(dagId, taskId, fn)`, + the Go body for a task Python declares with `@task.stub`. +4. **task_id is always written out**; nothing is derived from the Go function name. +5. **Every handler takes an `airflow.Context` first**, and nothing else is injected — a struct + embedding `context.Context`, exposing `Logger()`, `Client()`, `TaskInstance()`, and `DagRun()`. +6. **Every remaining parameter is data**, bound positionally, or by field when it is a single struct: + `arg:"..."` when tagged, else the folded Go field name. +7. This **breaks** `BundleProvider`, `Registry`, and `AddDag(dagId).AddTask(fn)`, with no + deprecation alias. + +## Context + +Python owns everything but the body of a Mixed Lang task: `@task.stub` declares the task, its +arguments, and its place in the graph. The Go side has no Dag to define, so Dag vocabulary misleads, +and it occupies the `AddDag` name that [ADR 7](0007-native-dag-interface.md) needs. + +Registration is inverted today. An author declares a struct with no state, asserts it implements +`v1.BundleProvider`, fills in `RegisterDags(dagbag v1.Registry) error`, and hands the struct to +`bundlev1server.Serve` — three concepts and an empty type before a single task is declared. The two +names are also one object: `Registry` is `Bundle` plus `AddDag`, the write side of the value that +later answers task lookups at execution time. + +The shipped signature then injects `sdk.TIRunContext`, `*slog.Logger`, and clients by type in any +position, so a handler can declare no context at all and its logger arrives separately from the +context it logs against. One required context carrying the rest fixes both. + +## Example + +```go +func main() { + bundle := airflow.Bundle() + + bundle.Register( + nativeEtl, // *airflow.DagRef, from ADR 7 + airflow.TaskHandler("py_etl", "transform", transform), + airflow.TaskHandler("py_etl", "via_struct_arg_tag", ViaStructArgTag), + ) + + if err := bundle.Serve(); err != nil { + log.Fatal(err) + } +} +``` + +Registration can be spread across packages, either by passing the bundle along or by returning +`[]airflow.Registration` for the caller to splat: `bundle.Register(taskflowbinding.Handlers()...)`. + +Three ways a Go function receives a stub task's data, all live in `go-sdk/example/bundle/`. + +**Flat positional**, for `def transform(country: str, extracted: dict)`: + +```go +func transform(actx airflow.Context, country string, extracted map[string]any) error { + actx.Logger().Info("transforming", "country", country, "try", actx.TaskInstance().TryNumber) + + threshold, err := actx.Client().GetVariable(actx, "etl_threshold") + if err != nil { + return err + } + return writeRows(actx, extracted, threshold) +} +``` + +**A single `arg:`-tagged struct**, for `def via_struct_arg_tag(region_code: str, threshold: float)`: + +```go +type ViaStructArgTagInput struct { + Region string `arg:"region_code"` + Threshold float64 `arg:"threshold"` +} + +func ViaStructArgTag(actx airflow.Context, input ViaStructArgTagInput) (any, error) +``` + +**A single untagged struct**, where the field name folds to the Python argument: `RegionCode` +lowercased with underscores stripped is `regioncode`, which matches `region_code`. + +## Consequences + +- `BundleProvider`, the empty provider struct, and the `bundlev1server` package all disappear from an + author's `main`; `Serve` becomes a method on the value they already hold. +- **Registration closes when `Serve` is called.** Registering afterwards is a programming error and + panics, like every other registration-time check in these ADRs. +- Every existing call site changes, acceptable only because the SDK's README already warns its APIs + "may change between releases without notice." +- Requiring `airflow.Context` turns a class of mistake into a compile error: a test calling + `myTask(context.Background(), ...)` no longer builds, where a package-level `airflow.Logger(ctx)` + accessor over a plain context would have compiled and then failed at run time on a missing value. +- Graceful termination needs no unwrapping — `actx.Done()` fires on supervisor shutdown, and + `http.NewRequestWithContext(actx, ...)` accepts it — while cleanup that must outlive cancellation + uses `context.WithoutCancel(actx)`. +- A helper typed as a plain `context.Context` recovers the methods with + `airflow.FromContext(ctx) (airflow.Context, bool)`. + +## Appendix: Implementation Notes + +- **One verb, over a sealed interface.** `Bundle.Register(items ...airflow.Registration)` accepts + both kinds because `*airflow.DagRef` and `airflow.TaskHandlerRef` satisfy `Registration`, an Review Comment: Renamed as `Registraterable`. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. + +## Decision + +1. **One Dag type, constructed then registered.** `airflow.Dag(spec)` returns a `*airflow.DagRef` + that is complete before `bundle.Register(dag)` takes it — the same verb that registers Mixed Lang + task handlers ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Naming rule: + `airflow.X(...)` constructs, `*airflow.XRef` is the handle. +2. **Tasks register through `dag.Task(fn any, opts ...airflow.TaskOption)`**, returning a + `*airflow.TaskRef`. `airflow.Inputs(...)` and a bare `airflow.TaskSpec{}` both satisfy + `TaskOption`, which is how one variadic carries both. +3. **`airflow.Inputs(refs...)` is data and an edge at once**, binding positionally after the context. +4. **`Before` and `After` are order-only edges on `airflow.Node`**, which both `*airflow.TaskRef` + and `*airflow.TaskGroupRef` satisfy, so a task and a whole group are equally an edge endpoint. + They are the Go pair for `>>` and `<<`, both variadic so a single call fans out. +5. **Trigger rules belong to the task**, as `airflow.TaskSpec{TriggerRule: ...}`, never to an edge. +6. **Everything an author writes comes from one `airflow` package.** +7. **No Go-native deferral**, and none is needed: the constructs that defer are DSL tasks Python + executes. + +## Context + +A native Dag is authored entirely in Go — schedule, tasks, and dependencies — and serializes into the +Dag JSON a Python Dag would produce. There is one Dag type, as in Python; the Mixed Lang case +registers task handlers instead ([ADR 6](0006-mixed-lang-task-handler-interface.md)) because it +defines no Dag. Dependencies between Go functions have to be typed rather than looked up by task ID, +and a Dag should read like Go rather than transliterated Python. The proposed interface spread its +surface across `v1`, `sdk`, and `slog`, published a half-built Dag to the registry and mutated it +afterwards, and could declare an edge in only one direction. + +## Example + +Both forms build the same graph; which one an author writes depends on whether the edge carries a +value. + +**Data dependencies — the TaskFlow equivalent.** `airflow.Inputs` passes an upstream's return value +in and declares the edge in one call, as calling one TaskFlow function with another's output does in +Python (`extracted = extract(); transformed = transform(extracted); load(transformed)`). + +```go +dag := airflow.Dag(airflow.DagSpec{DagId: "etl", Schedule: "@daily"}) Review Comment: Moved the dag_id as the positional argument, and added the signature as well. ########## go-sdk/adr/0007-native-dag-interface.md: ########## @@ -0,0 +1,148 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# 7. Native Dag interface + +Date: 2026-09-07 + +## Status + +Proposed. Supersedes the interfaces in #67155 and #70158, reshaped by the design review on #72043. +Nothing below exists on `main`. + +## Decision + +1. **One Dag type, constructed then registered.** `airflow.Dag(spec)` returns a `*airflow.DagRef` + that is complete before `bundle.Register(dag)` takes it — the same verb that registers Mixed Lang + task handlers ([ADR 6](0006-mixed-lang-task-handler-interface.md)). Naming rule: + `airflow.X(...)` constructs, `*airflow.XRef` is the handle. +2. **Tasks register through `dag.Task(fn any, opts ...airflow.TaskOption)`**, returning a + `*airflow.TaskRef`. `airflow.Inputs(...)` and a bare `airflow.TaskSpec{}` both satisfy + `TaskOption`, which is how one variadic carries both. +3. **`airflow.Inputs(refs...)` is data and an edge at once**, binding positionally after the context. +4. **`Before` and `After` are order-only edges on `airflow.Node`**, which both `*airflow.TaskRef` + and `*airflow.TaskGroupRef` satisfy, so a task and a whole group are equally an edge endpoint. + They are the Go pair for `>>` and `<<`, both variadic so a single call fans out. +5. **Trigger rules belong to the task**, as `airflow.TaskSpec{TriggerRule: ...}`, never to an edge. +6. **Everything an author writes comes from one `airflow` package.** +7. **No Go-native deferral**, and none is needed: the constructs that defer are DSL tasks Python + executes. + +## Context + +A native Dag is authored entirely in Go — schedule, tasks, and dependencies — and serializes into the +Dag JSON a Python Dag would produce. There is one Dag type, as in Python; the Mixed Lang case +registers task handlers instead ([ADR 6](0006-mixed-lang-task-handler-interface.md)) because it +defines no Dag. Dependencies between Go functions have to be typed rather than looked up by task ID, +and a Dag should read like Go rather than transliterated Python. The proposed interface spread its +surface across `v1`, `sdk`, and `slog`, published a half-built Dag to the registry and mutated it +afterwards, and could declare an edge in only one direction. + +## Example + +Both forms build the same graph; which one an author writes depends on whether the edge carries a +value. + +**Data dependencies — the TaskFlow equivalent.** `airflow.Inputs` passes an upstream's return value +in and declares the edge in one call, as calling one TaskFlow function with another's output does in +Python (`extracted = extract(); transformed = transform(extracted); load(transformed)`). + +```go +dag := airflow.Dag(airflow.DagSpec{DagId: "etl", Schedule: "@daily"}) + +extracted := dag.Task(extract) +transformed := dag.Task(transform, airflow.Inputs(extracted)) +dag.Task(load, airflow.Inputs(transformed), airflow.TaskSpec{Retries: 2}) + +bundle.Register(dag) +``` + +```go +func extract(actx airflow.Context) (Result, error) { + return Result{Message: "native Dag data"}, nil +} + +func transform(actx airflow.Context, extracted Result) (Result, error) { + return Result{Message: "transformed " + extracted.Message}, nil +} + +func load(actx airflow.Context, transformed Result) error { return nil } +``` + +**Order-only dependencies — the `>>` and `<<` equivalent.** For tasks that must be ordered but +exchange no data; the functions take no parameter for such an edge. + +```go +loaded := dag.Task(load, airflow.Inputs(transformed)) +cleaned := dag.Task(cleanup, airflow.TaskSpec{TriggerRule: airflow.AllDone}) +staging := dag.TaskGroup("staging") // a group carries edges like a task + +loaded.Before(dag.Task(notify), cleaned) // loaded >> [notify, cleanup] +cleaned.After(extracted) // cleanup << extracted +staging.Before(loaded) // staging >> load Review Comment: The shape of the label in Go SDK will be like `a.before(airflow.Label(b, "to b label"), airflow.Label(c, "to c label")`. This is equivalent to `a >> Label("to b label") >> b; a >> Label("to c label") >> c`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
