Thanks, TP. I like the general direction here. I really like the language-neutral abstraction.
Vikram On Mon, Oct 5, 2026 at 5:36 AM Kaxil Naik <[email protected]> wrote: > Thanks TP, I am looking forward to this. The design is sensible to me, I > added few comments on the PR around the code though, nothing major. > > > On Mon, 5 Oct 2026 at 10:40, Pierre Jeambrun <[email protected]> > wrote: > > > Thanks for opening the thread TP. > > > > This sounds like a nice plan forward, (proposed schema makes sense to me > > and delaying some advanced features while we focus on the MVP seems > > reasonable). > > > > I left a few small comments on the PR. > > > > Looking forward to hearing from other stakeholders. > > > > Pierre > > > > On Mon, Oct 5, 2026 at 8:00 AM Tzu-ping Chung via dev < > > [email protected]> wrote: > > > > > I thought through the compatibility_date vs $schema differences and I > > > think I tend to agree that using $schema would be easier for editors. > > > compatibility_date has an edge on user ergonomics, but these days > people > > > are probably always doing copy-paste or just using an agent to generate > > the > > > skeleton anyway. I’ll switch to $schema and drop compatibility_date. > > > > > > The parser PR already contains a schema file, and we’ll publish it > > similar > > > to API specs under airflow.apache.org <http://airflow.apache.org/>. > > > > > > TP > > > > > > > > > > On 3 Oct 2026, at 16:36, Shahar Epstein <[email protected]> wrote: > > > > > > > > Thanks for sharing it, TP! > > > > I support the overall direction, specifically the language neutral > > > format. > > > > For the format to work well with editors and other tools, could we > use > > a > > > > stable format identifier alongside/instead of compatibility_date, and > > > > document the schema and semantics in a written spec? It would also > help > > > to > > > > define metadata and typed params in the schema, reserve names for > > > deferred > > > > features such as assets and task groups, and clarify how run: tasks > map > > > to > > > > code when the YAML is read independently. > > > > I’d recommend looking at existing open workflow formats for > > inspiration. > > > > For example, the Open Workflow Specification > > > > <https://open-workflow-specification.org/> is more data-centric than > > > > task-centric, so it may not fit Airflow’s execution model, but its > > schema > > > > design could offer useful ideas. > > > > > > > > > > > > Shahar > > > > > > > > On Fri, Oct 2, 2026 at 2:04 PM Tzu-ping Chung via dev < > > > > [email protected]> wrote: > > > > > > > >> Hi all, > > > >> > > > >> With AIP-85, I've been working on a YAML Dag format, and would like > > some > > > >> feedback on it. Shortest possible taste first: > > > >> > > > >> compatibility_date: "2026-10-30" > > > >> dag_id: daily_sales > > > >> schedule: "@daily" > > > >> tasks: > > > >> # A classical task using an operator. > > > >> # There's no automatic template detection; {{ ds }} not wrapped > > in > > > >> $t is a literal. > > > >> - id: extract > > > >> uses: > > > >> > > > > > > airflow.providers.amazon.aws.transfers.s3_to_redshift.S3ToRedshiftOperator > > > >> with: > > > >> s3_bucket: retail-raw > > > >> s3_key: {$t: "sales/{{ ds }}.csv"} > > > >> schema: public > > > >> table: sales > > > >> # A taskflow-style task. > > > >> # Things inside run: are arguments to the function. > > > >> # $x means an XCom input. > > > >> - id: notify > > > >> run: > > > >> channel: "#data" > > > >> rows: {$x: extract} > > > >> > > > >> ## Why a new format > > > >> > > > >> DagFactory is the direct inspiration and the feature-parity bar; > > longer > > > >> term we'd like this to be the path that supersedes it. > > > >> > > > >> The reason to start fresh rather than extending is that DagFactory > is > > > >> essentially Python transcribed into YAML; import paths, callables > > named > > > by > > > >> file+function, a default_args block shaped like a DAG() call. With > > > recent > > > >> AIP-108 (language SDKs), YAML can provide a more neutral foundation > > for > > > >> declaring the dependency structure of tasks implemented in ANY > > > language. We > > > >> also want to better represent Airflow constructs such as XCom and > > > assets. > > > >> > > > >> == Design guidelines and decisions == > > > >> > > > >> - Language-neutral, not "Python in YAML". JSON is the data model, > YAML > > > >> just a skin: everything round-trips to JSON. > > > >> - Literal by default; templating is opt-in with {$t: ...} (inspired > by > > > >> AIP-80). This removes the implicit-Jinja surprises. Also, {$f: ...} > is > > > an > > > >> explicit file template, removing the classic `cat {{ ds }}.sh` -> > > > >> TemplateNotFound foot-gun. (There's also a $const to mark something > as > > > >> literal explicitly.) > > > >> - No default_args. Reusable `templates:` composed per task via > > > `extends:`, > > > >> and the merged task is validated against the real operator. An > > > unaccepted > > > >> argument is a parse error, not a silent drop. > > > >> - uses: a fully-qualified operator import path. run: the single code > > > >> primitive for any language (Python/Go/Java written identically). > > > >> - compatibility_date (borrowed from Cloudflare Workers) pins the > > format > > > >> semantics, so old files keep their meaning as the format evolves. > > > >> - It ships as an AIP-85 importer in the Task SDK, natively > recognized > > by > > > >> Airflow (enabled by a configuration). > > > >> > > > >> == Deliberately left out of v1 (all additive, can land later) == > > > >> > > > >> - Task groups, dynamic task mapping, branching; AIP-104/111/113 > > > (batching, > > > >> loops, dynamic task groups). > > > >> - Assets / data-aware scheduling (inlets/outlets, asset expressions, > > > >> watchers). > > > >> - Callbacks. They name a callable, so they wait on a declarative > > > >> callable-reference grammar. > > > >> - Python run: execution. The format is settled; the @task_handler > > > runtime > > > >> that runs it is a separate work (Go/Java run: tasks already > execute). > > > >> - Operator shorthand (e.g. amazon.S3ToRedshiftOperator) and > cross-file > > > >> shared config. > > > >> > > > >> I've also created a draft PR that implements the parser. (Not the > full > > > >> AIP-85 importer yet.) > > > >> https://github.com/apache/airflow/pull/74084 > > > >> > > > >> Thanks, > > > >> TP > > > >> > > > >> > > > >> > --------------------------------------------------------------------- > > > >> To unsubscribe, e-mail: [email protected] > > > >> For additional commands, e-mail: [email protected] > > > >> > > > >> > > > > > > > > >
