dabla commented on code in PR #62922: URL: https://github.com/apache/airflow/pull/62922#discussion_r4221130080
########## task-sdk/src/airflow/sdk/definitions/_internal/expandinput.py: ########## @@ -41,6 +42,84 @@ OperatorExpandKwargsArgument = Union["XComArg", Sequence[Union["XComArg", Mapping[str, Any]]]] +class Resolved(NamedTuple): + """ + An expand input resolved for iteration: its item count and an async read by index. + + The ``.iterate()`` counterpart of ``ExpandInput.resolve``, which hands one mapped task + instance the item at its ``map_index``: every source is pulled once, and ``aget(index)`` then + picks the item an index maps to the way ``resolve`` does for ``.expand()`` (the cross product + of ``iterate(**kwargs)``, the mapping at that position for ``iterate_kwargs``). Reads run on + the task's event loop next to the sub-tasks' own SDK calls, so nothing here may block the loop + thread on the supervisor channel (see ``AsyncAwareExecutor.imap_unordered``). + """ + + length: int + aget: Callable[[int], Awaitable[Mapping[str, Any]]] + + +class Source: + """ + One resolved expand argument, read by index without a blocking supervisor call on the loop. + + A Mapping is read as its ``(key, value)`` pairs and a scalar as a one-item sequence, matching + ``DictOfListsExpandInput._expand_mapped_field``. A value with async accessors (an + ``XComIterable``, a mapped upstream's ``LazyXComSequence``) is read through them. Anything + else may block in ``__getitem__`` (a ``.map()`` result over a lazy sequence, for instance), so + it is read in a worker thread. + """ + + @classmethod + async def from_argument(cls, argument: Any, context: Mapping[str, Any]) -> Source: + """Build a source from an expand argument: pull it if it is an XComArg, then make it a sequence.""" + if isinstance(argument, XComArg): + argument = await argument.aresolve(context) + if isinstance(argument, Mapping): + return cls(list(argument.items())) + if isinstance(argument, (str, bytes)) or not isinstance(argument, Iterable): Review Comment: Reproduced: a string from upstream ran as a single item. fb998e48fe checks the resolved XComArg in `Source.from_argument` with the same `is_mappable_value` the push check uses, raising `UnmappableXComTypePushed`, and `XComForMappingNotPushed` for `None`, as the upstream would have for a mapped dependant. Tests cover strings, bytes, scalars and `None` on `Source`, and `.iterate()` over an upstream returning a JSON string, which now fails before any item runs. Drafted-by: Claude Fable 5.1; reviewed by @dabla before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
