This is an automated email from the ASF dual-hosted git repository.
ash pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new c2c81525c05 Update readme docs for Go SDK (#56975)
c2c81525c05 is described below
commit c2c81525c05639087b946b23f7514752997c4e7d
Author: Ash Berlin-Taylor <[email protected]>
AuthorDate: Wed Oct 22 10:45:16 2025 +0100
Update readme docs for Go SDK (#56975)
This is still far from complete, but it is closer to being able to be put in
hands of users now
---
go-sdk/README.md | 114 +++++++++++++++++++++++++-----------------
go-sdk/bundle/bundlev1/doc.go | 2 +-
go-sdk/pkg/config/config.go | 2 +-
go-sdk/sdk/sdk.go | 3 --
4 files changed, 69 insertions(+), 52 deletions(-)
diff --git a/go-sdk/README.md b/go-sdk/README.md
index 3e94af2929b..179f0ba8c99 100644
--- a/go-sdk/README.md
+++ b/go-sdk/README.md
@@ -17,82 +17,103 @@
under the License.
-->
-# 🚧 Apache Airflow Go Task SDK 🚧
+# Apache Airflow Go Task SDK
+
+The Go SDK uses the Task Execution Interface (TEI or Task API) introduced in
AIP-72 with Airflow 3.0.0 to give
+Task functions written in Go full access to the Airflow "model", natively in
go.
+
+The Task API however does not provide a means to get the `ExecuteTaskWorkload`
to the go worker itself. For
+that we use the Edge Executor API.
+Longer term we will likely need to stabilize the Edge Executor API and add
versioning to it.
+
+Since Go is a compiled language (putting aside projects such as
[YAEGI](https://github.com/traefik/yaegi) that allow Go to be interpreted), all
tasks must:
+
+1. Be compiled into a binary ahead of time, and
+2. Be registered inside the worker process in order to be executed.
+
> [!NOTE]
> This Golang SDK is under active development and is not ready for prime-time
> yet.
-This README is primarily aimed at developers working _on_ the Go-SDK itself.
Users wishing to write Airflow tasks in Go should look at the reference docs,
but those don't exist yet.
+## Quickstart
-## How It Works
+- See [`example/bundle/main.go`](./example/bundle/main.go) for an example dag
bundle where you can define your task functions
-The Go SDK uses the Task Execution Interface (TEI or Task API) introduced in
AIP-72 with Airflow 3.0.0.
+- Compile this into a binary:
-The Task API however does not provide a means to get the `ExecuteTaskWorkload`
to the go worker itself. For the short term, we make use of
[gopher-celery](https://github.com/marselester/gopher-celery) to get tasks from
a Redis broker. Longer term we will likely need to stabilize the Edge Executor
API and write a go client for that.
+ ```bash
+ go build -o ./bin/sample-dag-bundle ./example/bundle
+ ```
-Since Go is a compiled language (putting aside projects such as
[YAEGI](https://github.com/traefik/yaegi) that allow go to be interpreted) all
tasks must be a) compiled in to the binary, and b) "registered" inside the
worker process in order to be executed.
+ (or see the [`Justfile`](./example/bundle/Justfile) for how you can build it
and specify they bundle version number at build time.)
-## Quick Testing Setup
+- Configure the go edge worker, by editing `$AIRFLOW_HOME/go-sdk.yaml`:
-The Go SDK currently works with Airflow's Celery Executor setup. Here's how to
get started:
+ These config values need tweaking, expecially the ports and secrets. The
ports are the default assuming
+ airflow is running locally via `airflow standalone`.
-### Prerequisites
+ ```toml
+ [edge]
+ api_url = "http://0.0.0.0:8080/";
-- Go 1.24 or later
-- Docker and Docker Compose (for Breeze)
-- Redis (for Celery broker)
+ [execution]
+ api_url = "http://0.0.0.0:8080/execution";
-### Step 1: Start Airflow with Celery Executor
+ [api_auth]
+ # This needs to match the value from the same setting in your API server for
Edge API to function
+ secret_key = "hPDU4Yi/wf5COaWiqeI3g=="
-Start Breeze with Celery executor:
+ [bundles]
+ # Which folder to look in for pre-compiled bundle binaries
+ folder = "./bin"
-```bash
-breeze start-airflow --backend postgres --executor CeleryExecutor
--load-example-dags
-```
+ [logging]
+ # Where to write task logs to
+ base_log_folder = "./logs"
+ # Secret key matching airflow API server config, to only allow log requests
from there.
+ secret_key = "u0ZDb2ccINAbhzNmvYzclw=="
+ ```
-This will start:
+ You can also set these options via environment variables of
`AIRFLOW__${section}_${key}`, for example `AIRFLOW__API_AUTH__SECRET_KEY`.
-- Airflow API Server on `http://localhost:28080`
-- Celery workers (we will not utilise this)
-- Redis broker on `localhost:26379`
-- Loads the example DAGs
+- Install the worker
-### Step 2: Stop the Celery Worker
+ ```bash
+ go install github.com/apache/airflow/go-sdk/cmd/airflow-go-edge-worker@latest
+ ```
-We want to run the go workers instead of running the Celery ones. So in
`breeze`, press CTRL+C to
-stop the Celery workers.
+- Run it!
-### Step 3: Run the Go SDK Worker
+ ```bash
+ airflow-go-edge-worker run --queues golang
+ ```
-From the `go-sdk` directory, run the example worker:
+### Example Dag:
-```bash
-go run ./example/main.go run \
- --broker-address=localhost:26379 \
- --queues default \
- --execution-api-url http://localhost:28080/execution
-```
+You will need to create a python Dag and deploy it in to the Airflow
-**Parameters explained:**
+```python
+from airflow.sdk import dag, task
-- `--broker-address=localhost:26379`: Redis broker address (default Celery
broker)
-- `--queues default`: Queue name where Celery enqueues tasks
-- `--execution-api-url http://localhost:28080/execution`: Airflow's Task
Execution API endpoint
[email protected](queue="golang")
+def extract():...
-### Step 4: Submit a Test Task
[email protected](queue="golang")
+def transform(): ...
-You can submit tasks through the Airflow UI for dag_id: `tutorial_dag`. The Go
worker will pick up tasks from the Celery queue and execute them using the Task
Execution Interface.
+@dag()
+def simple_dag():
-Observe the logs in the terminal where you run the test task.
+ extract() >> transform()
-## Current state
+multi_language()
+```
-This SDK currently will:
+Here we see the `@task.stub` which tells the Dag parser about the "shape" of
the go tasks, and lets us define
+the relationships between them
-- Get tasks from Celery queue(s)
-- Run registered tasks (no support for dag versioning or loading of multiple
different "bundles")
-- Heartbeat and report the final state of the final TI
-- Allow access to Variables
+> [!NOTE]
+> Yes, you still have to have a python Dag file for now. This is a known
limitation at the moment.
## Known missing features
@@ -100,7 +121,6 @@ A non-exhaustive list of features we have yet to implement
- Support for putting tasks into state other than success or
failed/up-for-retry (deferred, failed-without-retries etc.)
- Remote task logs (i.e. S3/GCS etc)
-- XCom reading/writing from API server
- XCom reading/writing from other XCom backends
diff --git a/go-sdk/bundle/bundlev1/doc.go b/go-sdk/bundle/bundlev1/doc.go
index 246bc9423cc..cedf6d31e36 100644
--- a/go-sdk/bundle/bundlev1/doc.go
+++ b/go-sdk/bundle/bundlev1/doc.go
@@ -17,7 +17,7 @@
// Package bundlev1 defines the interfaces and types need to be an Airflow Dag
Bundle
//
-// The main entrypoint should call [bundlev1server/Serve].
+// The main entry point should call [bundlev1server/Serve].
package bundlev1
// We call this package `.../bundle/bundlev1` (duplicating "bundle") so that
the uses of it in code are by
diff --git a/go-sdk/pkg/config/config.go b/go-sdk/pkg/config/config.go
index 6fb5c66f161..aedb80f9256 100644
--- a/go-sdk/pkg/config/config.go
+++ b/go-sdk/pkg/config/config.go
@@ -116,7 +116,7 @@ func SetupViper(cfgFile string) (*viper.Viper, error) {
airflowHome = path.Join(home, "airflow")
}
- // Search config in home directory with name ".go-sdk" (without
extension).
+ // Search config in AIRFLOW_HOME directory with name
viper.AddConfigPath(airflowHome)
viper.SetConfigType("yaml")
viper.SetConfigName("go-sdk.yaml")
diff --git a/go-sdk/sdk/sdk.go b/go-sdk/sdk/sdk.go
index a644b07cc78..679ab22d8f4 100644
--- a/go-sdk/sdk/sdk.go
+++ b/go-sdk/sdk/sdk.go
@@ -15,9 +15,6 @@
// specific language governing permissions and limitations
// under the License.
-/*
-Package sdk provides access to the Airflow objects (Variables, Connection,
XCom etc) during run time for tasks.
-*/
package sdk
import (
