This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 5d3e6362601 Improve Java SDK quick start setup (#72440)
5d3e6362601 is described below
commit 5d3e6362601ddb6f14a46554e61e045d3822013e
Author: Andrew Chang <[email protected]>
AuthorDate: Wed Sep 9 05:53:06 2026 +0800
Improve Java SDK quick start setup (#72440)
The quick start did not show a complete project layout or distinguish Dag
delivery from Java bundle deployment. This made the runnable setup and worker
requirements unclear.
---
.../language-sdks/java.rst | 208 ++++++++++++++++++---
1 file changed, 182 insertions(+), 26 deletions(-)
diff --git a/airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst
b/airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst
index a7424b4ecd3..4bae1652230 100644
--- a/airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst
+++ b/airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst
@@ -39,6 +39,8 @@ The generated API reference for the Java SDK is published
with the Airflow docum
Prerequisites
-------------
+* JDK 11 or later is required on the machine that builds the Java project. A
local Gradle installation is only
+ needed to generate the Gradle Wrapper for a new project.
* JRE 11 or later must be available on the Airflow worker nodes.
* The compiled task JAR(s) and JVM dependencies must be accessible from the
worker.
* The ``apache-airflow-task-sdk`` package (installed with Airflow) provides
the coordinator;
@@ -47,11 +49,22 @@ Prerequisites
Quick start
-----------
-The following example shows the minimal moving parts: a Python Dag with two
stub tasks, and a Java
-implementation of those tasks.
+This example uses the annotation-based API to build a Java bundle for two stub
tasks. The annotation-based
+and interface-based APIs are two ways to define Java tasks, not two bundle
formats. They use different Java
+code and dependencies, but the standard Gradle source layout, entry-point
requirement, ``bundle`` task, and
+deployment process are the same. See :ref:`java-sdk/interface-api` for the
interface-based task code.
-Python Dag (the scheduling side)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The Python Dag source and the Java Gradle project are independent. They do not
need to be in the same
+repository or have any particular relative filesystem layout. The Dag follows
the deployment's normal Dag
+delivery process; only the compiled Java bundle is deployed from the Gradle
project to ``jars_root``.
+
+Define the Python Dag
+~~~~~~~~~~~~~~~~~~~~~
+
+For a local installation with the default ``[core] dags_folder``, create
+``${AIRFLOW_HOME}/dags/sales_pipeline.py``. More generally, create
``sales_pipeline.py`` in the source location
+used by the deployment's normal Dag delivery process, such as its configured
Dags folder, Dag repository, or
+Dag bundle. This path is not relative to the Java Gradle project.
.. code-block:: python
@@ -75,31 +88,108 @@ Python Dag (the scheduling side)
sales_pipeline()
-Java implementation
-~~~~~~~~~~~~~~~~~~~
+Create the Java project
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The guide uses ``com.mycompany.airflow.sales`` as a placeholder Java package.
Replace it with the package for
+your project, and update the source paths, ``package`` declarations, and
``mainClass`` together.
+
+Start with this standard Gradle project layout. The ``gradlew`` scripts and
``gradle/wrapper/`` directory
+are generated by the `Gradle Wrapper
<https://docs.gradle.org/current/userguide/gradle_wrapper.html>`__.
+
+.. code-block:: text
+
+ sales-pipeline-java/
+ ├── build.gradle
+ ├── gradle.properties
+ ├── settings.gradle
+ ├── gradlew
+ ├── gradlew.bat
+ ├── gradle/
+ │ └── wrapper/
+ │ ├── gradle-wrapper.jar
+ │ └── gradle-wrapper.properties
+ └── src/
+ └── main/
+ └── java/
+ └── com/
+ └── mycompany/
+ └── airflow/
+ └── sales/
+ ├── Main.java
+ └── SalesPipeline.java
+
+Choose a Java SDK version published to
+`Maven Central
<https://central.sonatype.com/artifact/org.apache.airflow/airflow-sdk>`__, then
set it in
+``gradle.properties`` by replacing ``JAVA_SDK_VERSION``:
+
+.. code-block:: properties
+
+ airflowJavaSdkVersion=JAVA_SDK_VERSION
+
+The Java SDK is experimental, so Maven Central may list only prerelease
versions. Select a version deliberately
+rather than copying a version that may become stale in this guide.
+
+Name the project in ``settings.gradle``:
+
+.. code-block:: groovy
+
+ rootProject.name = "sales-pipeline"
+
+Configure the build in ``build.gradle``:
+
+.. code-block:: groovy
+
+ plugins {
+ id("org.apache.airflow.sdk") version "${airflowJavaSdkVersion}"
+ }
+
+ repositories {
+ mavenCentral()
+ }
+
+ dependencies {
+
annotationProcessor("org.apache.airflow:airflow-sdk-processor:${airflowJavaSdkVersion}")
+
implementation("org.apache.airflow:airflow-sdk:${airflowJavaSdkVersion}")
+ }
+
+ java {
+ toolchain {
+ languageVersion.set(JavaLanguageVersion.of(11))
+ }
+ }
+
+ airflowBundle {
+ mainClass = "com.mycompany.airflow.sales.Main"
+ }
+
+The ``annotationProcessor`` dependency is required for this annotation-based
example. Omit it when using the
+interface-based API. If the project does not have the Gradle Wrapper yet,
generate it from the project root
+with ``gradle wrapper``.
+
+Implement the Java tasks
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Save the task implementations as
+``src/main/java/com/mycompany/airflow/sales/SalesPipeline.java``:
.. code-block:: java
- import org.apache.airflow.sdk.*;
+ package com.mycompany.airflow.sales;
+
+ import org.apache.airflow.sdk.Builder;
@Builder.Dag(id = "sales_pipeline")
public class SalesPipeline {
@Builder.Task(id = "extract")
- public long extract(Client client) {
- var conn = client.getConnection("sales_db");
- // ... fetch data using conn.host, conn.login, conn.password ...
- return recordCount;
+ public long extract() {
+ return 3;
}
@Builder.Task(id = "transform")
- public long transform(
- Client client,
- @Builder.XCom(task = "extract") long recordCount
- ) {
- var threshold = (String) client.getVariable("transform_threshold");
- // ... process data ...
- return transformedCount;
+ public long transform(@Builder.XCom(task = "extract") long recordCount) {
+ return recordCount * 2;
}
}
@@ -108,15 +198,25 @@ Java implementation
See how both ``transform`` in Python and Java need to have an argument to
accept upstream XCom. The
Python one is needed to declare dependency, and the Java one is needed to
actually retrieve the value.
-Java entry point
-~~~~~~~~~~~~~~~~
+Add the Java entry point
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Save the entry point as
``src/main/java/com/mycompany/airflow/sales/Main.java``:
.. code-block:: java
+ package com.mycompany.airflow.sales;
+
+ import java.util.List;
+
+ import org.apache.airflow.sdk.BundleBuilder;
+ import org.apache.airflow.sdk.DagDef;
+ import org.apache.airflow.sdk.Server;
+
public class Main implements BundleBuilder {
@Override
public Iterable<DagDef> getDags() {
- return List.of(SalesPipelineBuilder.build()); // SalesPipelineBuilder
generated at compile time
+ return List.of(SalesPipelineBuilder.build());
}
public static void main(String[] args) {
@@ -124,21 +224,68 @@ Java entry point
}
}
-Coordinator configuration
-~~~~~~~~~~~~~~~~~~~~~~~~~
+``SalesPipelineBuilder`` is generated by the annotation processor during
compilation.
+
+Build and deploy
+~~~~~~~~~~~~~~~~
+
+Run the bundle task from the ``sales-pipeline-java/`` project root:
+
+.. code-block:: bash
+
+ ./gradlew bundle
+
+The deployable JAR or JARs are now in ``build/bundle/``. Copy or mount that
entire directory into a path
+available on every worker that can consume the ``java`` queue. For example,
deploy it as
+``/opt/airflow/jars/sales-pipeline/``. Java source files do not need to be
deployed to Airflow.
+
+For a local deployment where those paths are writable, the copy commands could
be:
+
+.. code-block:: bash
+
+ mkdir -p /opt/airflow/jars/sales-pipeline
+ cp build/bundle/*.jar /opt/airflow/jars/sales-pipeline/
+
+Deploy ``sales_pipeline.py`` separately through the deployment's normal Dag
delivery process. For example,
+that process might sync it to ``${AIRFLOW_HOME}/dags/`` or package it in a Dag
bundle; neither location is
+inside or relative to ``sales-pipeline-java/``.
+
+Configure Airflow so the coordinator scans the parent JAR directory
recursively and routes the ``java`` queue
+to it. Add the following ``[sdk]`` section to the file selected by
``AIRFLOW_CONFIG`` (by default,
+``${AIRFLOW_HOME}/airflow.cfg``), or set the equivalent ``AIRFLOW__SDK__*``
environment variables:
.. code-block:: ini
[sdk]
coordinators = {
- "java-jdk17": {
+ "java": {
"classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
"kwargs": {"jars_root": ["/opt/airflow/jars"]}
}
}
- queue_to_coordinator = {"java": "java-jdk17"}
+ queue_to_coordinator = {"java": "java"}
+
+``java`` is a user-chosen coordinator name, not a reserved value. The value
assigned to the queue in
+``queue_to_coordinator`` must match a key in ``coordinators``.
+
+Restart the affected Airflow components after changing this configuration. The
coordinator config and JARs
+must be available wherever tasks execute. With ``CeleryExecutor``, that means
the Celery workers; with
+``LocalExecutor``, tasks run in subprocesses on the scheduler's host. The API
server and Dag processor do not
+need the JARs, while the Dag processor must receive ``sales_pipeline.py``
through the separate Dag delivery
+process.
+
+After Airflow has parsed the Dag, trigger it from the UI or command line:
+
+.. code-block:: bash
+
+ airflow dags trigger sales_pipeline
+
+The Java ``extract`` task returns ``3``, ``transform`` returns ``6`` through
XCom, and the Python ``load`` task
+logs ``Loaded: 6``.
See :ref:`java-sdk/coordinator-config` for the full list of accepted
``kwargs``.
+For a larger example that exercises connections, variables, logging, and both
task APIs, see the
+`Java SDK example in the Airflow repository
<https://github.com/apache/airflow/tree/main/java-sdk/example>`__.
Writing tasks
-------------
@@ -251,8 +398,17 @@ nested ``static`` class like ``ProcessTask``:
.addTask("process", ProcessTask.class);
return List.of(dag);
}
+
+ public static void main(String[] args) {
+ Server.create(args).serve(new MyBundle().build());
+ }
}
+Place the task classes and ``BundleBuilder`` under the standard
``src/main/java/<package>/`` source tree.
+The ``BundleBuilder`` can provide the ``main`` method itself, as above, or a
separate entry-point class can
+call it. Set ``airflowBundle.mainClass`` to the class that provides ``main``.
From that point onward, both APIs
+use the same ``./gradlew bundle`` command and deploy the resulting
``build/bundle/`` directory in the same way.
+
See the `Java SDK API Reference
<https://airflow.apache.org/docs/java-sdk/stable/>`__ for more details.
.. _java-sdk/logging:
@@ -374,7 +530,7 @@ its level) and installs ``AirflowJulHandler`` in their
place:
public static void main(String[] args) {
AirflowJulHandler.setup();
- Server.create(args).serve(new MyBundle());
+ Server.create(args).serve(new MyBundle().build());
}
Alternatively, declare the handler in a ``logging.properties`` file and point
JUL at it with the
