jk-shah opened a new issue, #41373:
URL: https://github.com/apache/beam/issues/41373

   ### What happened?
   
   In `apache_beam.yaml.yaml_mapping` 
(`sdks/python/apache_beam/yaml/yaml_mapping.py`, lines `331-368` and `419-432`) 
and `apache_beam.yaml.json_utils` 
(`sdks/python/apache_beam/yaml/json_utils.py`, lines `72-111`), nullable 
`output_type` declarations on `MapToFields` fail at runtime or schema parsing 
time in two ways:
   
   1. **`_validator(beam_type)` ignores `beam_type.nullable` 
(`yaml_mapping.py:331-368`)**:
      Even when `json_schema_to_beam_schema` (`json_utils.py:77`) marks an 
optional `object` property as `nullable=True` (omitted from `required`), 
`_validator` runs `isinstance(x, str)` (etc.) without allowing `None`. 
Returning `Row(a=None)` for an optional property raises `TypeError: Row(a=None) 
violates schema ...` inside `checking_func` (`yaml_mapping.py:428-429`).
   2. **`json_type_to_beam_type` has no nullable syntax for top-level 
`output_type` (`json_utils.py:83-111`)**:
      `json_type_to_beam_type` always constructs `schema_pb2.FieldType` with 
`nullable=False`, ignores `nullable: true`, and crashes with `TypeError: 
unhashable type: 'list'` at line `88` (`if type_name in 
JSON_ATOMIC_TYPES_TO_BEAM:`) when standard JSON Schema nullable syntax `type: 
['string', 'null']` is passed. As a result, a `MapToFields` callable/expression 
returning `Optional[str]` cannot declare `output_type` when `None` values are 
present.
   
   Minimal reproducer:
   
   ```python
   import apache_beam as beam
   from apache_beam.portability.api import schema_pb2
   from apache_beam.yaml import json_utils
   from apache_beam.yaml import yaml_mapping
   
   # 1) Optional object property (nullable=True) rejected when None:
   schema = json_utils.json_schema_to_beam_schema(
       {'type': 'object', 'properties': {'a': {'type': 'string'}}})
   assert schema.fields[0].type.nullable is True
   v = yaml_mapping._validator(
       schema_pb2.FieldType(row_type=schema_pb2.RowType(schema=schema)))
   print(v(beam.Row(a=None)))  # False (expected True)
   
   # 2) MapToFields callable returning None with output_type raises TypeError:
   fn = yaml_mapping._as_callable(
       ['x'],
       {'callable': 'lambda r: None', 'output_type': 'string'},
       'MapToFields',
       'python',
       {'x': str})
   fn(beam.Row(x='hi'))  # TypeError: None violates schema {'type': 'string'}
   ```
   
   ### Cause
   
   `_validator(beam_type)` dispatches on `beam_type.WhichOneof("type_info")` 
without short-circuiting `if beam_type.nullable and x is None: return True`, 
and `json_type_to_beam_type` does not honor `nullable: true` or `type: 
['<atomic>', 'null']` via `maybe_nullable(...)`.
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [x] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [x] Component: Beam YAML
   - [ ] Component: Beam ML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Infrastructure
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Prism Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to