potiuk commented on code in PR #69750: URL: https://github.com/apache/airflow/pull/69750#discussion_r3564403462
########## airflow-core/docs/img/diagram_task_sdk_execution_sequence.py: ########## @@ -0,0 +1,286 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# /// script +# requires-python = ">=3.10" +# dependencies = [ +# "rich>=13.6.0", +# "graphviz>=0.20.1", +# ] +# /// +""" Review Comment: Done — this is now a true sequence diagram. Each participant (Executor, Task, Supervisor, Execution API) has its own lifeline, with the Supervisor placed in the middle so the Task ↔ Supervisor round-trip (steps 7→10: `GetConnection/GetVariable/GetXCom` → proxied Execution-API call → `*Result` back to the task) reads as arrows going back and forth between the two lifelines. Arrows are straight and the step descriptions are margined so they stay clear of the lines. Pushed in b5331899d5. --- Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting ########## airflow-core/docs/img/diagram_java_sdk_execution_architecture.py: ########## @@ -0,0 +1,287 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# /// script +# requires-python = ">=3.10" +# dependencies = [ +# "rich>=13.6.0", +# "graphviz>=0.20.1", +# ] +# /// +""" +Architecture diagram for the Java (JVM) Task SDK. + +Unlike the Go SDK (a standalone edge worker), the Java SDK plugs into the *same* +Python Supervisor via a new **Coordinator** layer: + +* ``CoordinatorManager`` resolves the task's ``queue`` to a ``BaseCoordinator`` + (``JavaCoordinator`` for the ``java`` queue, ``_PythonCoordinator`` otherwise); +* ``JavaCoordinator.execute_task()`` opens two loopback-TCP servers, spawns a JVM + bundle process with ``subprocess.Popen``, and drives it with + ``_JavaActivitySubprocess`` — a subclass of the shared ``ActivitySubprocess``; +* the JVM process connects *back* over TCP and speaks the same msgpack protocol as + a Python task, so the Python side heartbeats, proxies every Execution-API call, + and manages state. The JVM task therefore **never holds the task JWT**. + +Rendered with graphviz directly so labels sit inside sized shapes: 3-D box = +native OS process, rounded box = an object inside a process, component = a server +app, cylinder = the database, note = a caption. +""" + +from __future__ import annotations + +from pathlib import Path + +import graphviz +from rich.console import Console + +MY_DIR = Path(__file__).parent +MY_FILENAME = Path(__file__).with_suffix("").name + +console = Console(width=400, color_system="standard") + Review Comment: Done — the Java execution flow (`diagram_java_sdk_execution_sequence.py`) is now a true sequence diagram: Scheduler, JVM subprocess, Supervisor and Execution API each on their own lifeline, Supervisor central, so the JVM ↔ Supervisor round-trip over loopback TCP (steps 9→12) shows as request/response arrows going back and forth. The architecture diagram stays as the static structure and the round-trip now lives in the sequence diagram. Pushed in b5331899d5. --- Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting ########## airflow-core/docs/core-concepts/overview.rst: ########## @@ -192,6 +192,88 @@ code is never executed in the context of the *scheduler*. bundle version when dispatching each task. If needed, the cadence of sync and scan of the *Dag bundle* can be configured. +Task execution architecture +--------------------------- + +The diagrams above show how Airflow's components are *deployed*. The diagrams below instead show what happens +*inside a worker when a task actually runs* — how the Task SDK, the Supervisor and Coordinator processes, and +the language runtimes work together: which processes are involved, and the classes and protocols they use to +communicate. + +.. _overview-task-sdk-execution-architecture: + +Python Task SDK execution +......................... + +When a *worker* actually runs a task, it does not run the user's code directly. Instead it starts a +lightweight **Supervisor** that runs in its own **native operating-system process** and +*forks* a second native process in which the **Task SDK** runtime (``task_runner``) executes the user code. +The two processes talk over a socket, and the Supervisor is the only side that ever holds the short-lived +task JWT or talks to the *Execution API* — the user's code never sees the token and never touches the +database. + +The same runtime can also run *in-process* (a single Python process, no fork, no sockets, no HTTP) for +``dag.test()`` and local runs. The diagram below contrasts the two paths and marks where each Python process +lives: + +.. image:: ../img/diagram_task_sdk_execution_architecture.png + +The message flow of a supervised run — startup, running the user code, proxied Connection/Variable/XCom +lookups, heartbeats, and reporting the final state — is shown below as a numbered, top-to-bottom sequence. +Each step names the process that performs it, and each arrow is coloured by its sender and labelled with the +message class or protocol used: + +.. image:: ../img/diagram_task_sdk_execution_sequence.png + +.. _overview-non-python-language-sdks: + +Non-Python language SDKs (Go and Java) +...................................... + +The Task Execution Interface (TEI) introduced in AIP-72 is language-agnostic, so a task can also be written in +a **compiled, non-Python language**. A Python Dag still declares the task with ``@task.stub(queue=...)`` (so +Python and non-Python tasks can be mixed in one Dag), but the actual work is delegated to the matching runtime. +There are currently **two different integration styles** — the Go SDK runs a standalone worker, while the Java +SDK plugs into the existing Python Supervisor. + +.. _overview-go-sdk-architecture: + +**Go SDK — standalone edge worker.** The `Go Task SDK Review Comment: Sounds good — I've left the Go-SDK-first vs Coordinator-first ordering to your follow-up as you offered, since it's mainly your change and you can verify it yourself. Not touched in this update. --- Drafted-by: Claude Code (Opus 4.8); reviewed by @potiuk before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
