Thanks TP, I am looking forward to this. The design is sensible to me, I added few comments on the PR around the code though, nothing major.
On Mon, 5 Oct 2026 at 10:40, Pierre Jeambrun <[email protected]> wrote: > Thanks for opening the thread TP. > > This sounds like a nice plan forward, (proposed schema makes sense to me > and delaying some advanced features while we focus on the MVP seems > reasonable). > > I left a few small comments on the PR. > > Looking forward to hearing from other stakeholders. > > Pierre > > On Mon, Oct 5, 2026 at 8:00 AM Tzu-ping Chung via dev < > [email protected]> wrote: > > > I thought through the compatibility_date vs $schema differences and I > > think I tend to agree that using $schema would be easier for editors. > > compatibility_date has an edge on user ergonomics, but these days people > > are probably always doing copy-paste or just using an agent to generate > the > > skeleton anyway. I’ll switch to $schema and drop compatibility_date. > > > > The parser PR already contains a schema file, and we’ll publish it > similar > > to API specs under airflow.apache.org <http://airflow.apache.org/>. > > > > TP > > > > > > > On 3 Oct 2026, at 16:36, Shahar Epstein <[email protected]> wrote: > > > > > > Thanks for sharing it, TP! > > > I support the overall direction, specifically the language neutral > > format. > > > For the format to work well with editors and other tools, could we use > a > > > stable format identifier alongside/instead of compatibility_date, and > > > document the schema and semantics in a written spec? It would also help > > to > > > define metadata and typed params in the schema, reserve names for > > deferred > > > features such as assets and task groups, and clarify how run: tasks map > > to > > > code when the YAML is read independently. > > > I’d recommend looking at existing open workflow formats for > inspiration. > > > For example, the Open Workflow Specification > > > <https://open-workflow-specification.org/> is more data-centric than > > > task-centric, so it may not fit Airflow’s execution model, but its > schema > > > design could offer useful ideas. > > > > > > > > > Shahar > > > > > > On Fri, Oct 2, 2026 at 2:04 PM Tzu-ping Chung via dev < > > > [email protected]> wrote: > > > > > >> Hi all, > > >> > > >> With AIP-85, I've been working on a YAML Dag format, and would like > some > > >> feedback on it. Shortest possible taste first: > > >> > > >> compatibility_date: "2026-10-30" > > >> dag_id: daily_sales > > >> schedule: "@daily" > > >> tasks: > > >> # A classical task using an operator. > > >> # There's no automatic template detection; {{ ds }} not wrapped > in > > >> $t is a literal. > > >> - id: extract > > >> uses: > > >> > > > airflow.providers.amazon.aws.transfers.s3_to_redshift.S3ToRedshiftOperator > > >> with: > > >> s3_bucket: retail-raw > > >> s3_key: {$t: "sales/{{ ds }}.csv"} > > >> schema: public > > >> table: sales > > >> # A taskflow-style task. > > >> # Things inside run: are arguments to the function. > > >> # $x means an XCom input. > > >> - id: notify > > >> run: > > >> channel: "#data" > > >> rows: {$x: extract} > > >> > > >> ## Why a new format > > >> > > >> DagFactory is the direct inspiration and the feature-parity bar; > longer > > >> term we'd like this to be the path that supersedes it. > > >> > > >> The reason to start fresh rather than extending is that DagFactory is > > >> essentially Python transcribed into YAML; import paths, callables > named > > by > > >> file+function, a default_args block shaped like a DAG() call. With > > recent > > >> AIP-108 (language SDKs), YAML can provide a more neutral foundation > for > > >> declaring the dependency structure of tasks implemented in ANY > > language. We > > >> also want to better represent Airflow constructs such as XCom and > > assets. > > >> > > >> == Design guidelines and decisions == > > >> > > >> - Language-neutral, not "Python in YAML". JSON is the data model, YAML > > >> just a skin: everything round-trips to JSON. > > >> - Literal by default; templating is opt-in with {$t: ...} (inspired by > > >> AIP-80). This removes the implicit-Jinja surprises. Also, {$f: ...} is > > an > > >> explicit file template, removing the classic `cat {{ ds }}.sh` -> > > >> TemplateNotFound foot-gun. (There's also a $const to mark something as > > >> literal explicitly.) > > >> - No default_args. Reusable `templates:` composed per task via > > `extends:`, > > >> and the merged task is validated against the real operator. An > > unaccepted > > >> argument is a parse error, not a silent drop. > > >> - uses: a fully-qualified operator import path. run: the single code > > >> primitive for any language (Python/Go/Java written identically). > > >> - compatibility_date (borrowed from Cloudflare Workers) pins the > format > > >> semantics, so old files keep their meaning as the format evolves. > > >> - It ships as an AIP-85 importer in the Task SDK, natively recognized > by > > >> Airflow (enabled by a configuration). > > >> > > >> == Deliberately left out of v1 (all additive, can land later) == > > >> > > >> - Task groups, dynamic task mapping, branching; AIP-104/111/113 > > (batching, > > >> loops, dynamic task groups). > > >> - Assets / data-aware scheduling (inlets/outlets, asset expressions, > > >> watchers). > > >> - Callbacks. They name a callable, so they wait on a declarative > > >> callable-reference grammar. > > >> - Python run: execution. The format is settled; the @task_handler > > runtime > > >> that runs it is a separate work (Go/Java run: tasks already execute). > > >> - Operator shorthand (e.g. amazon.S3ToRedshiftOperator) and cross-file > > >> shared config. > > >> > > >> I've also created a draft PR that implements the parser. (Not the full > > >> AIP-85 importer yet.) > > >> https://github.com/apache/airflow/pull/74084 > > >> > > >> Thanks, > > >> TP > > >> > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: [email protected] > > >> For additional commands, e-mail: [email protected] > > >> > > >> > > > > >
