uranusjr commented on code in PR #77080:
URL: https://github.com/apache/airflow/pull/77080#discussion_r4627279823


##########
airflow-core/adr/yaml-dags/0001-task-mapping-authoring-interface.md:
##########
@@ -0,0 +1,211 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+# ADR-0001: Task Mapping — YAML Authoring Interface
+
+## Status
+
+Proposed.
+
+## Context
+
+What a mapped task is, and the rules every authoring surface follows, are in 
[Language SDK ADR-0016](../lang-sdk/0016-mapped-tasks-across-lang-sdks.md). 
This ADR records their spelling in YAML. `map` is deferred since it requires a 
way to reference a callable (see Decision 4 below).
+
+YAML has no method calls to chain. ADR-0016 makes `zip` and `concat` methods 
on a reference, so they chain in declaration order. YAML nests them instead. 
Since YAML already marks a value in place (`{$x: ...}` for an upstream output), 
an expanded argument is marked the same way:
+
+```yaml
+tasks:
+  # Returns e.g. [1, 2, 3]
+  - id: list_files
+    run: {}
+
+  # Equivalent to process.partial(bucket="raw").expand(path=list_files)
+  - id: process
+    run:
+      bucket: raw
+      path: {$expand: {$x: list_files}}
+
+  # Reducing; equivalent to summarize(sizes=process)
+  - id: summarize
+    run:
+      sizes: {$x: process}
+```
+
+## Decision
+
+1. **An expanded argument is marked in place with `$expand`. Every unmarked 
argument is a `partial()`.**
+   - For example, `path: {$expand: <source>}` under `with:` or `run:` is 
Python's `.expand(path=<source>)`.
+   - Several `$expand` arguments give a cross product, as several `expand()` 
keywords do. It is not allowed to nest `$expand` inside a construct.
+
+2. **Expansion is possible on a literal list, a `$x`, or one of `$zip` and 
`$concat`.**
+   - A list gives one instance per item. Its items are literal values; a 
marker inside it is an error. Any other literal is an error.
+   - `{$x: task}` expands over the task's return value. When `task` is itself 
mapped, the instances line up one-to-one by map index. `$x` with a `key` is 
rejected, as in Python.

Review Comment:
   Reworded Decision 2 to one instance per non-null result, in map-index order, 
and fixed ADR-0016’s read-path row in this PR too.
   
   A language-SDK producer can store a null row (#77281 aligns that across 
SDKs), and leaving those out of the sequence reads is that PR’s “Nulls in 
mapped results” follow-up, which ADR-0016 now links to.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to