Hello everyone,
Airflow Dags are no longer Python only. With the language SDKs you can now
author Dags in Java, Go and TypeScript: declare the whole Dag (schedule,
tasks, options and dependencies) in that language, or implement the stub
tasks of a Python Dag in it.
I'd like to put the focus on TypeScript. The TypeScript SDK ships with
Airflow 3.4 (release candidates are out) as the @apache-airflow/ts-sdk npm
package, and it is still in beta. I'd love to get community feedback on it.
A minimal Dag looks like this:
import { Bundle, Dag } from "@apache-airflow/ts-sdk";
const dag = new Dag("ts_hello", { schedule: "@daily", queue:
"typescript" });
const extract = dag.task("extract", async (): Promise<number> => 42);
const report = dag.task("report", async ({ rows }: { rows: number }) =>
{
console.log(`extracted ${rows} rows`);
});
report({ rows: extract() });
await new Bundle(dag).serve();
Calling a task with another task's result draws the dependency, and task
inputs are type checked, so a misspelled argument is a compile error.
`npx airflow-ts-pack src/main.ts` builds the Dag into one bundle file, which
goes into a Dag bundle like any other Dag file.
Not everything is supported yet. Task groups, if/switch branching, edge
labels, triggerDagRun and human-in-the-loop work today. Task loops, dynamic
task mapping, Dag params, callbacks, assets and `dags test` are among the
things still to come. The compatibility matrix in the docs tracks it for
each SDK.
I opened a GitHub issue to collect the feedback in one place, with
screenshots of the code, the local setup and the UI, and a few questions we
would most like answers to:
https://github.com/apache/airflow/issues/76370
Please comment there (or reply here) with what feels natural, what feels
awkward, and what you would need first. Short snippets of the code you would
like to write are welcome.
Links:
- TypeScript SDK guide:
https://airflow.staged.apache.org/docs/apache-airflow/3.4.0/authoring-and-scheduling/language-sdks/typescript.html
- API reference: https://airflow.staged.apache.org/docs/ts-sdk/stable/
- Compatibility matrix:
https://airflow.staged.apache.org/docs/apache-airflow/3.4.0/authoring-and-scheduling/language-sdks/index.html#compatibility-matrix
- Examples: https://github.com/apache/airflow/tree/main/ts-sdk/example
(The 3.4 docs links are on the staged site until 3.4.0 is released.)
Thanks,
Pierre
