jason810496 commented on code in PR #69750:
URL: https://github.com/apache/airflow/pull/69750#discussion_r3564094902
##########
airflow-core/docs/core-concepts/overview.rst:
##########
@@ -192,6 +192,88 @@ code is never executed in the context of the *scheduler*.
bundle version when dispatching each task. If needed, the cadence of sync
and scan
of the *Dag bundle* can be configured.
+Task execution architecture
+---------------------------
+
+The diagrams above show how Airflow's components are *deployed*. The diagrams
below instead show what happens
+*inside a worker when a task actually runs* — how the Task SDK, the Supervisor
and Coordinator processes, and
+the language runtimes work together: which processes are involved, and the
classes and protocols they use to
+communicate.
+
+.. _overview-task-sdk-execution-architecture:
+
+Python Task SDK execution
+.........................
+
+When a *worker* actually runs a task, it does not run the user's code
directly. Instead it starts a
+lightweight **Supervisor** that runs in its own **native operating-system
process** and
+*forks* a second native process in which the **Task SDK** runtime
(``task_runner``) executes the user code.
+The two processes talk over a socket, and the Supervisor is the only side that
ever holds the short-lived
+task JWT or talks to the *Execution API* — the user's code never sees the
token and never touches the
+database.
+
+The same runtime can also run *in-process* (a single Python process, no fork,
no sockets, no HTTP) for
+``dag.test()`` and local runs. The diagram below contrasts the two paths and
marks where each Python process
+lives:
+
+.. image:: ../img/diagram_task_sdk_execution_architecture.png
+
+The message flow of a supervised run — startup, running the user code, proxied
Connection/Variable/XCom
+lookups, heartbeats, and reporting the final state — is shown below as a
numbered, top-to-bottom sequence.
+Each step names the process that performs it, and each arrow is coloured by
its sender and labelled with the
+message class or protocol used:
+
+.. image:: ../img/diagram_task_sdk_execution_sequence.png
+
+.. _overview-non-python-language-sdks:
+
+Non-Python language SDKs (Go and Java)
+......................................
+
+The Task Execution Interface (TEI) introduced in AIP-72 is language-agnostic,
so a task can also be written in
+a **compiled, non-Python language**. A Python Dag still declares the task with
``@task.stub(queue=...)`` (so
+Python and non-Python tasks can be mixed in one Dag), but the actual work is
delegated to the matching runtime.
+There are currently **two different integration styles** — the Go SDK runs a
standalone worker, while the Java
+SDK plugs into the existing Python Supervisor.
+
+.. _overview-go-sdk-architecture:
+
+**Go SDK — standalone edge worker.** The `Go Task SDK
Review Comment:
For Go-SDK I prefer to mention the Coordinator approach first, then the edge
worker one. Since the edge worker lack of the following feature this section
pointed out --
https://github.com/apache/airflow/tree/main/go-sdk#known-limitations
Additionally, the Java, Go, upcoming TS SDK all go with the Coordinator as
first-class direction.
I can do this in follow-up as those are mainly my change and I'm able to
verify myself.
##########
airflow-core/docs/img/diagram_task_sdk_execution_sequence.py:
##########
@@ -0,0 +1,286 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+# "rich>=13.6.0",
+# "graphviz>=0.20.1",
+# ]
+# ///
+"""
Review Comment:
Ditto for the Task (Or perhaps task runner more accurately) <-> Supervisor
part. How about having the round trip part as sequence diagram.
##########
airflow-core/docs/img/diagram_java_sdk_execution_architecture.py:
##########
@@ -0,0 +1,287 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+# "rich>=13.6.0",
+# "graphviz>=0.20.1",
+# ]
+# ///
+"""
+Architecture diagram for the Java (JVM) Task SDK.
+
+Unlike the Go SDK (a standalone edge worker), the Java SDK plugs into the
*same*
+Python Supervisor via a new **Coordinator** layer:
+
+* ``CoordinatorManager`` resolves the task's ``queue`` to a ``BaseCoordinator``
+ (``JavaCoordinator`` for the ``java`` queue, ``_PythonCoordinator``
otherwise);
+* ``JavaCoordinator.execute_task()`` opens two loopback-TCP servers, spawns a
JVM
+ bundle process with ``subprocess.Popen``, and drives it with
+ ``_JavaActivitySubprocess`` — a subclass of the shared
``ActivitySubprocess``;
+* the JVM process connects *back* over TCP and speaks the same msgpack
protocol as
+ a Python task, so the Python side heartbeats, proxies every Execution-API
call,
+ and manages state. The JVM task therefore **never holds the task JWT**.
+
+Rendered with graphviz directly so labels sit inside sized shapes: 3-D box =
+native OS process, rounded box = an object inside a process, component = a
server
+app, cylinder = the database, note = a caption.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import graphviz
+from rich.console import Console
+
+MY_DIR = Path(__file__).parent
+MY_FILENAME = Path(__file__).with_suffix("").name
+
+console = Console(width=400, color_system="standard")
+
Review Comment:
From user point of view, perhaps we could make the JVM <-> Supervisor part
as sequence diagram to showcase the round-trip to make it more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
