jason810496 opened a new pull request, #73970: URL: https://github.com/apache/airflow/pull/73970
Stack (bottom to top): **#73970**, #73971, #73972, #73973, #73974, #73975, #73976, #73977 Language SDK coordinators now read their compiled task handlers only from a Dag bundle, as [ADR-0013](https://github.com/apache/airflow/blob/main/airflow-core/adr/lang-sdk/0013-persisted-task-handler-bindings.md) decides. This is the first layer of the ADR-0013 stack. - `jars_root` (Java) and `executables_root` (Go / native), which shipped in 3.3 as experimental, are removed without a shim, as is the unreleased `bundles_root` (TypeScript). A config that still sets one fails with `Cannot instantiate coordinator '<key>'` on the first task routed to it. No newsfragment is added, since the Language SDKs are marked experimental. - `dag_bundle_name` from #70805 is renamed `task_handler_bundle_name`. A later native-Dag change needs `dag_bundle_name` with a different meaning, and it has not shipped in a release, so there is no shim. - With `task_handler_bundle_name` set, the task uses that Dag bundle at its version current when the task starts. Unset, it uses the task's own Dag bundle at the run's version. For a versioned Dag bundle, the version is pinned for the whole task, as in #70805. An unversioned one such as `LocalDagBundle` is read in place. - The name is validated when `[sdk] coordinators` is loaded, for every entry, without constructing the coordinator. Loading happens at every task start, so a typo in any entry fails every task on that worker, Python tasks included, as a bad `queue_to_coordinator` key already does. KubernetesExecutor also loads it in the scheduler, and a bad name makes it ignore the `extra` of every coordinator with a warning. - How handlers are found inside the Dag bundle is unchanged. Later layers replace that scan with a binding persisted at parse time. Before: ```ini [sdk] coordinators = {"jdk-17": {"classpath": "airflow.sdk.coordinators.java.JavaCoordinator", "kwargs": {"jars_root": ["/opt/airflow/jars"]}}} ``` After: ```ini [dag_processor] dag_bundle_config_list = [ {"name": "dags-folder", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {}}, {"name": "java-task-handlers", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/opt/airflow/jars"}} ] [sdk] coordinators = {"jdk-17": {"classpath": "airflow.sdk.coordinators.java.JavaCoordinator", "kwargs": {"task_handler_bundle_name": "java-task-handlers"}}} ``` The artifact Dag bundle is registered in `dag_bundle_config_list` on every component, like any other Dag bundle, and that registration is what the worker resolves the name through. The docs cover the expected layout (artifacts in their own Dag bundle), the fallback to the task's own Dag bundle, that one Java artifact bundle is one classpath (conflicting dependency versions need a second bundle, coordinator and queue), and that Go artifacts need a Dag bundle that keeps the executable bit, which object-store bundles such as `S3DagBundle` do not. The compose e2e and the k8s lang-SDK harness are migrated to artifact Dag bundles. **Known cost:** JARs in a registered Dag bundle are listed by the Dag processor as Dag files (a JAR is a zip), so each gets a no-op parse per interval. Packed Go binaries and `*.min.mjs` files are not listed. A later layer addresses it. **Known limitation:** a worker resolving any Dag bundle imports every configured bundle class, so a worker running a coordinator task needs the providers of all bundles in `dag_bundle_config_list`, as it already does for Python tasks. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
