uranusjr commented on code in PR #65956: URL: https://github.com/apache/airflow/pull/65956#discussion_r3322937300
########## java-sdk/README.md: ########## @@ -0,0 +1,101 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +# Airflow Java SDK + +A **JVM** SDK for Apache Airflow. You can use any JVM-compatible language to write +workflow bundles, and have Airflow consume the result. + +The SDK and execution-time logic is implemented in Kotlin. +An example is bundled showing how the SDK can be used in Java. + +## Building the SDK + +```bash +./gradlew build +``` + +## Running the example + +* Put the [DAG with stub tasks](./dags) to somewhere Airflow can find. + +* Ensure the `java` command is available in the same environment the Airflow + task worker is in. + +* Package the example and its dependencies into JARs in + `./example/build/install/example/lib` + + ```bash + ./gradlew :example:installDist + ``` + +* Configure Airflow to route tasks in the *java* queue to be run with Java: + + ```bash + export AIRFLOW__SDK__COORDINATORS='{ + "java": { + "classpath": "airflow.sdk.coordinators.java.JavaCoordinator", + "kwargs": {"jars_root": ["/opt/airflow/java-sdk/example/build/install/example/lib"]} + } + }' + export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java"}' + ``` + +* Ensure the Connection and Variable needed by the example DAG are available: + + ```bash + export AIRFLOW_CONN_TEST_HTTP='{ + "conn_type": "http", + "login": "user", + "password": "pass", + "host": "example.com", + "port": 1234, + "extra": {"param1": "val1", "param2": "val2"} + }' + export AIRFLOW_VAR_MY_VARIABLE=123 + ``` + +## Technical Details + +The user uses the SDK to implement a Java application that implements task +methods, and metadata on which DAG and task each method should be used +for. + +When the Airflow Supervisor identifies a task should be run with Java, it +launches the Java application as a subprocess. The Java application accepts +flags `--comm` and `--logs` from the command line to identify TCP sockets it +should connect to, and communicates with the Supervisor through these channels +during execution. + +1. On connection, the Supervisor immediately sends a StartupDetails message + through the comm socket. +2. The Java application finds and executes the relevant method. +3. During execution, the Java application uses the comm socket to retrieve + information (e.g. Variable) from, and send data (e.g. XCom) to Airflow. +4. The Java application informs the comm socket to tell the Supervisor the + task's terminal state. +5. The Java application exits. + +During the Java application's lifetime, it also sends log messages generated by +the SDK (not user code) through the logs socket, so the Supervisor can append +them to Airflow logs. + +Communication uses the same formats as the Python-based processes. + Review Comment: Coroutine is not a thing in Java… -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
