Thanatos9404 commented on code in PR #41201: URL: https://github.com/apache/beam/pull/41201#discussion_r4535835319
########## sdks/python/apache_beam/typehints/arrow_type_compatibility.py: ########## @@ -295,13 +304,71 @@ def _arrow_type_from_beam_fieldtype( return output_arrow_type +def _from_arrow_value_fn(typ) -> Optional[Callable[[Any], Any]]: + """Returns a function that turns a value from ``pa.Scalar.as_py`` into + ``typ``, or None if the value needs no conversion. + + ``as_py`` returns structs as dicts, so nested rows are built from those + dicts, including rows in optional fields and in lists. + """ + if isinstance(typ, RowTypeConstraint): + return _row_from_dict_fn(typ.user_type, typ._fields) + if match_is_named_tuple(typ): + return _row_from_dict_fn( + typ, [(name, typ.__annotations__[name]) for name in typ._fields]) + if match_dataclass_for_row(typ): + return _row_from_dict_fn( + typ, [(field.name, field.type) for field in dataclasses.fields(typ)]) + + origin = get_origin(typ) + args = get_args(typ) + if origin is Union: Review Comment: The first case was real: with `batch_type=pa.Array`, `Optional[Address]` and `List[Address]` elements still came out as dicts because `from_typehints` normalizes the hint. a42a60f handles the normalized union and sequence constraints and adds a test for both. I matched the constraints directly since `convert_to_python_type` raises for `Collection[...]`. A top-level `Address | None` is normalized to the same union, so it is covered too; PEP 604 unions inside a row field fail earlier in schema conversion, which this PR does not touch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
