Lee-W commented on code in PR #74695: URL: https://github.com/apache/airflow/pull/74695#discussion_r4374365349
########## providers/common/ai/src/airflow/providers/common/ai/utils/output_type.py: ########## @@ -38,6 +46,70 @@ _CONVERSION_ERRORS = (ValidationError, ValueError, TypeError, AttributeError, KeyError, IndexError) +def is_model_dump_required(output: Any, output_type: Any, *, serialize_output: bool) -> bool: + """ + Tell whether an output holding ``BaseModel`` instances must be pushed to XCom dumped. + + The instances are looked up in the output itself, including the elements of a list, tuple, set or + frozenset and the values of a dict. A downstream task can load a model from XCom only if the worker + registered the model's class, and the worker registers the classes ``iter_pydantic_models`` finds in + ``output_type``. It finds none in a list of output types, an output marker, an output function, a + ``TypeAliasType`` or a ``NewType``, so an instance of a class it did not find is dumped. + Without ``iter_pydantic_models`` there is no evidence either way, so the caller's ``serialize_output`` decides. + """ + if not (found := [model for model, _ in _iter_models_in_output(output)]): + return False + if serialize_output: + return True + if iter_pydantic_models is None: + return False + registered = set(iter_pydantic_models(output_type)) + return any(type(model) not in registered for model in found) + + +def dump_output_for_xcom(output: Any, output_type: Any, *, serialize_output: bool) -> Any: + """ + Return the output as it is pushed to XCom: dumped when ``is_model_dump_required``, else unchanged. + + With ``serialize_output`` the dump holds Python values, as the ``model_dump()`` of 1.0.0 does, so a + ``date`` or ``Decimal`` field keeps its type; an output with a set, frozenset or dict key holding a model + is dumped to JSON values instead. A model the worker registers no class for, without ``serialize_output``, + is dumped to JSON values as well, field names rather than aliases, so a ``HttpUrl``, ``Path`` or ``Enum`` + field is a string that XCom accepts. + """ + if not is_model_dump_required(output, output_type, serialize_output=serialize_output): + return output + # Python mode is what 1.0.0 pushed, a minor release must not change it. A model dumped to a dict cannot + # be a set member, so python mode raises on a set holding models. + if serialize_output and not any(in_set for _, in_set in _iter_models_in_output(output)): + if isinstance(output, BaseModel): + return output.model_dump() + return TypeAdapter(Any).dump_python(output, mode="python") Review Comment: `dump_output_for_xcom` now calls `model_dump()` only when the output itself is a model and `serialize_output=True`, as in 1.0.0. A `list[Link]` or `dict[str, Link]` of registered models is left to serde, and tests push one through `serialize`. Only a model the worker cannot register is dumped, to JSON values. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
