uranusjr commented on code in PR #74698:
URL: https://github.com/apache/airflow/pull/74698#discussion_r4344381350
##########
airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst:
##########
@@ -548,123 +628,154 @@ surface as Go values when read back via ``GetXCom``.
.. note::
- ``GetXCom`` returns the value exactly as decoded from the transport; there
is no typed XCom
- deserialization layer yet. The Python supervisor encodes values as
``msgpack``, so a whole number arrives
- as a Go integer type (whose width depends on the value) and only a
non-integer arrives as ``float64``. Do
- not assume a fixed integer width: type-switch over the numeric types you
expect, or round-trip the value
- through ``json.Marshal`` / ``json.Unmarshal`` into a typed Go value.
+ An integer arrives as a Go integer type whose width depends on the value.
Type-switch over the integer types
+ you expect, or round-trip the value through ``json.Marshal`` and
``json.Unmarshal`` into a typed Go value. A
+ task input from ``airflow.Inputs`` or a TaskFlow argument is already decoded
into its parameter type.
.. _go-sdk/build:
-Building and packaging
-----------------------
-
-A plain ``go build`` produces a runnable binary, but a *deployable* bundle
(binary + embedded source files +
-manifest) must be produced with ``airflow-go-pack``. The packer compiles the
bundle and appends the embedded
-metadata footer, so the coordinator can read its ``dag_id``\ s without
executing the binary, producing a
-single runnable file. The on-disk format the packer emits (the ``AFBNDL01``
footer and the
-``airflow-metadata.yaml`` manifest) is the bundle format shared by all
native-executable SDKs, specified in
-:doc:`task-sdk:executable-bundle-spec`.
-
-``airflow-go-pack`` ships via the Go 1.24 ``tool`` directive, so there is no
global install: add
-
-.. code-block:: text
-
- tool github.com/apache/airflow/go-sdk/cmd/airflow-go-pack
-
-to your bundle module's ``go.mod`` and run it with ``go tool
airflow-go-pack``. This pins the packer version
-per project.
+Building a bundle
+-----------------
-Build and pack in one step; any flags after ``--`` are forwarded verbatim to
``go build``:
+``airflow-go-pack`` builds your ``main`` package into a single executable, the
bundle. It also embeds the
+entrypoint file and the file that declares each Dag, which the Airflow UI's
**Code** view shows, and a manifest
+of the bundle's Dags and task handlers. That file is all you deploy, and
Airflow refuses to run a bundle whose
+content was changed after it was built.
.. code-block:: bash
- go tool airflow-go-pack ./example/bundle -- -trimpath -tags=prod
+ go tool airflow-go-pack --output dist/my_dags ./cmd/my_dags -- -trimpath
-Use ``--output <path>`` to write the packed bundle straight into the directory
of the Dag bundle the
-coordinator scans (see `Deploying`_):
+* ``--output <path>`` sets the output file. By default, the bundle is written
to the current directory, named
+ after the package's directory.
+* Flags after ``--`` are passed to ``go build``.
+* ``go tool airflow-go-pack inspect <bundle>`` prints a bundle's manifest.
-.. code-block:: bash
+The Dag processor reads a bundle only when its file name has no extension,
such as ``dags/orders``, not
+``dags/orders.bin``.
- go tool airflow-go-pack --output
/opt/airflow/go-task-handlers/sample-dag-bundle ./example/bundle
+.. _go-sdk/cross-platform:
Cross-platform builds
-~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~
-The worker and the Dag processor that run a bundle often use a different
operating system or CPU
-architecture than your build machine (for example, deploying to a Linux host
from an Apple-silicon
-``darwin/arm64`` laptop). Pass ``--goos`` / ``--goarch`` and the packer
cross-builds for you:
+The workers and the Dag processor often run a different operating system or
CPU architecture than your build
+machine. Pass ``--goos`` and ``--goarch`` to build for them:
.. code-block:: bash
- go tool airflow-go-pack --goos linux --goarch amd64 \
- --output /opt/airflow/go-task-handlers/sample-dag-bundle \
- ./example/bundle
+ go tool airflow-go-pack --goos linux --goarch amd64 --output dist/my_dags
./cmd/my_dags
-Alternatively, pack a pre-built binary with ``--executable`` / ``--source``.
The packer normally execs the
-binary with ``--airflow-metadata`` to read its manifest, but a cross-compiled
binary cannot run on the build
-host. In that case, generate the manifest on a machine that *can* run the
binary and feed it to the packer
-with ``--airflow-metadata``:
+``airflow-go-pack`` runs the built binary to read its manifest. When it cannot
do that for a cross-compiled
+binary, build and read the manifest on a machine of the target platform, then
pack the binary with
+``--executable``:
.. code-block:: bash
# On a linux/amd64 machine:
- go build -o my-bundle ./example/bundle
- ./my-bundle --airflow-metadata > airflow-metadata.yaml
+ go build -o my_dags ./cmd/my_dags
+ ./my_dags --airflow-metadata > airflow-metadata.yaml
- # Back on the darwin/arm64 machine:
- go tool airflow-go-pack --executable ./my-bundle --source main.go \
- --airflow-metadata airflow-metadata.yaml
-
-(``--executable`` is mutually exclusive with ``--goos`` / ``--goarch`` and
with ``go build`` flags after
-``--``, since it packs an already-built binary instead of building one.)
+ # On the build machine:
+ go tool airflow-go-pack --executable ./my_dags --source
cmd/my_dags/main.go \
+ --airflow-metadata airflow-metadata.yaml --output dist/my_dags
Deploying
-~~~~~~~~~
+---------
-Copy or mount the packed bundle into the Dag bundle named by the coordinator's
``task_handler_bundle_name``.
-The :class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator` scans
that Dag bundle recursively,
-matches the incoming ``dag_id`` against each bundle's manifest, verifies the
bundle's integrity hash, and
-launches the matching bundle. This scan identifies bundles by the trailer
magic, not by file name. The Dag
-processor also requires a file name without an extension, see
:ref:`go-sdk/native-dag-parsing`.
+Where the bundle goes depends on what it contains.
-The matching bundle is marked executable before it is launched, so any Dag
bundle works, including an
-object-store one such as ``S3DagBundle`` that has no concept of file
permissions and so cannot preserve
-the execute bit the build produced.
+Dag definition
+~~~~~~~~~~~~~~
-.. _go-sdk/native-dag-parsing:
+Put the bundle in a Dag bundle, such as the default ``dags-folder`` bundle.
The Dag processor runs each bundle it
+finds to read its Dags, as it reads any other Dag file, so the Dag processor
needs the same ``[sdk]``
+configuration as the workers, and a bundle built for its platform:
-Parsing native Dags
-~~~~~~~~~~~~~~~~~~~
+.. code-block:: ini
+
+ [sdk]
+ coordinators = {"go": {"classpath":
"airflow.sdk.coordinators.executable.ExecutableCoordinator"}}
+ queue_to_coordinator = {"golang": "go"}
-Once an :class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator` is
configured, the Dag processor
-parses every bundle binary in each Dag bundle and runs it to collect the Dags
it defines. The Dag processor
-recognizes a bundle binary only when its file name has no extension, such as
``bin/orders``, not
-``bin/orders.bin``. A binary that only registers task handlers is parsed too:
each parse runs it and finds
-no Dags, and a binary built with a Go SDK that cannot answer the Dag-parse
request records an import error.
-Keep those out of the Dag processor with ``.airflowignore``, as described in
:ref:`Quick start <go-sdk/quick-start>`.
+With more than one ``ExecutableCoordinator``, map each Dag bundle that holds
Go Dags to one of them in
+``[sdk] dag_bundle_to_coordinator``, such as ``{"dags-folder": "go"}``, or its
bundles fail to parse.
-With one :class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator`,
it parses the binaries of every Dag
-bundle. With several, for example a second one for another
``task_handler_bundle_name``, map each Dag bundle that
-holds native Go Dags to one of them in ``[sdk] dag_bundle_to_coordinator``. A
binary in a Dag bundle that has no
-entry, or whose entry names no
:class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator`, fails to
parse
-with an import error:
+Their tasks run from the same Dag bundle, at the version their Dag run was
created with, so a Dag declared in Go
+needs no coordinator option to find its bundle.
+
+A bundle that cannot be read, or whose Dags fail a check, shows up as an
import error, and the other Dags keep
+working.
+
+Python Dag with stub TaskHandler
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, the coordinator loads the bundle that implements a stub task from
the stub task's own Dag bundle, at
+the version its Dag run was created with, so the bundle goes next to the
Python Dag that declares the stub.
+
+To keep such bundles in a Dag bundle of their own, name it with
``task_handler_bundle_name``. A task then uses the
+version that bundle is on when the task starts:
.. code-block:: ini
+ [dag_processor]
+ dag_bundle_config_list = [
+ {"name": "dags-folder", "classpath":
"airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {}},
+ {
+ "name": "go-task-handlers",
+ "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle",
+ "kwargs": {"path": "/opt/airflow/go-task-handlers"}
+ }
+ ]
+
[sdk]
- dag_bundle_to_coordinator = {"dags-folder": "go"}
+ coordinators = {
+ "go": {
+ "classpath":
"airflow.sdk.coordinators.executable.ExecutableCoordinator",
+ "kwargs": {"task_handler_bundle_name": "go-task-handlers"}
+ }
+ }
+ queue_to_coordinator = {"golang": "go"}
+
+The coordinator finds the bundle for a task by the ``dag_id`` in each bundle's
manifest, not by file name, so one
+Dag bundle can hold several bundles. Any Dag bundle type works, including an
object-store one such as
+``S3DagBundle``.
+
+The Dag processor checks the stub tasks of each Python Dag against the task
handlers the bundles register, so
+it needs the bundles too, built for its platform.
-* The Code view shows each native Dag's own source file, taken from the
sources the bundle embeds.
-* A task of a native Go Dag runs the bundle binary its Dag was parsed from.
+The Dag processor also parses a bundle that only implements stub tasks, which
runs it once per parse and finds
+no Dags. Skip that with the Dag bundle's ``.airflowignore``; the coordinator
still finds the bundles. For a Dag
+bundle that holds only such bundles, as ``go-task-handlers`` above:
+
+.. code-block:: bash
+
+ # [core] dag_ignore_file_syntax = glob (the default)
+ echo '*' > /opt/airflow/go-task-handlers/.airflowignore
+
+ # [core] dag_ignore_file_syntax = regexp
+ echo '.' > /opt/airflow/go-task-handlers/.airflowignore
+
+For bundles next to a Python Dag, put them in a folder such as ``bin/`` and
ignore it with ``bin/*`` (glob) or
+``^bin/`` (regexp).
+
+Where the configuration is needed
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The coordinator runs where tasks run, so the ``[sdk]`` configuration and the
bundles are needed on the workers.
+With ``CeleryExecutor``, set them on the Celery workers. With
``LocalExecutor``, tasks run in the scheduler's
+process, so set them there. The Dag processor needs them too, as described
above. Register every Dag bundle in
Review Comment:
Same as the Java PR, this is not accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
