jason810496 opened a new pull request, #74037: URL: https://github.com/apache/airflow/pull/74037
Replaces #73844, which GitHub closed as merged into a stack branch when the stack was reordered. Stack (bottom to top): #73841, #74004, #73842, #73843, **#73844**, #73845, #73846, #73847 Part of the native Dag e2e stack, related to #71929. Builds on the Java importer layer ([new commits only](https://github.com/apache/airflow/compare/jason/lang-sdk-e2e/04-java-dag-importer...jason/lang-sdk-e2e/05-node-dag-importer)). Merge after #73442 and #73445, which make the TypeScript runtime answer the Dag-parsing request and add `triggerDagRun`. ## Why The earlier layers let a coordinator parse native Dags with its own runtime, and `JavaCoordinator` opts in. This makes `NodeCoordinator` opt in, so Airflow parses the Dags declared in packed `*.min.mjs` TypeScript bundles. ## What changes A `NodeCoordinator` without `bundles_root` now parses the TypeScript bundles in the Dag bundles it serves, and the same entry runs their tasks: ```ini [sdk] coordinators = { "ts": { "classpath": "airflow.sdk.coordinators.node.NodeCoordinator", "kwargs": {"node_executable": "/usr/local/bin/node"} } } queue_to_coordinator = {"typescript": "ts"} ``` - Only files that end in `.min.mjs` and start with the header `airflow-ts-pack` writes are parsed, and `safe_mode` does not change this check. A file that cannot be read is kept, so its parse reports the error. - The parse runs `node <bundle>` with the supervisor schema version from the bundle metadata. A bundle that fails its integrity check gets an import error. - A coordinator with `bundles_root` still parses nothing, as before. - The TypeScript runtime is unchanged here. The TS SDK layer above makes it answer the parse request. - `typescript.rst` documents the setup, that the Dag processor needs Node.js, that a Dag with a cycle is an import error, and the Code view. ## Decision left open by #71929 #71929 does not settle what `get_source_code` returns for a native Dag. Here the Code view shows the bundle's entry module for each of its Dags, since a Dag's source is stored per file. A short notice replaces a source that cannot be read. ## Limitations - A bundle that only registers `TaskHandler` objects is parsed too when it sits in a served Dag bundle, because its metadata does not say whether it declares Dags. Each parse launches `node` and finds no Dags, so the docs suggest keeping such bundles under `bundles_root`. - `dag_policy` and `task_policy` do not run on a native Dag. `airflow dags test`, `tasks test`, `tasks render` and `tasks list` refuse it, and `airflow dags reserialize` does not store the Dags of `*.min.mjs` bundles. The docs say so. - The intro and the first Limitations bullet of `typescript.rst` still describe only stub Dags. This stack leaves them, since #73875 rewrites that page. ## How to test ```bash uv run --project task-sdk pytest task-sdk/tests/task_sdk/coordinators/node ``` The native e2e layer at the top of the stack, above the SDK layers, parses a native TypeScript Dag end to end and checks its source and a triggered run. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Claude Code Opus 5.5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
