jk-shah opened a new issue, #41101:
URL: https://github.com/apache/beam/issues/41101
### What happened?
`beam.io.WriteToParquet` (and Beam YAML `WriteToParquet`) fails when a
schema'd PCollection has a nested row field whose values are `beam.Row`
instances:
```
pyarrow.lib.ArrowTypeError: Could not convert Row(city='NYC', zip='10001')
with type Row: was not a dict, tuple, or recognized null value for conversion
to struct type
```
The schema itself is fine (it maps to `struct<city: string, zip: string>`);
only the values can't be converted. The same data typed as nested
`typing.NamedTuple`s writes fine, and Beam YAML `Create` with nested mappings
succeeds but pipelines that build nested rows with `beam.Row` (for example
after `ReadFromJson`/`ReadFromParquet` plus a `MapToFields` that keeps a struct
column) fail.
Minimal reproducer (Beam 2.75.0, Python 3.14; `produce_batch` below is
unchanged on `master` @ `728ef1b4`):
```python
import apache_beam as beam
with beam.Pipeline() as p:
(p
| beam.Create([1, 2])
| beam.Map(lambda i: beam.Row(id=i, addr=beam.Row(city='NYC',
zip='10001')))
| beam.io.WriteToParquet('/tmp/out', file_name_suffix='.parquet'))
# ArrowTypeError: Could not convert Row(city='NYC', zip='10001') with type
Row:
# was not a dict, tuple, or recognized null value for conversion to struct
type
```
### Cause
`WriteToParquet.expand` converts elements to `pa.Table` batches through
`PyarrowBatchConverter.produce_batch`
(`sdks/python/apache_beam/typehints/arrow_type_compatibility.py`):
```python
arrays = [
pa.array([getattr(el, name) for el in elements],
type=self._arrow_schema.field(name).type)
for name, _ in self._element_type._fields
]
```
Top-level fields are read with `getattr`, but nested values are passed to
`pa.array` as-is. pyarrow accepts dicts, tuples and None for struct types, but
`beam.Row` is none of these (it isn't a tuple subclass, unlike a NamedTuple),
so the conversion fails. Converting nested `beam.Row` values (recursively,
including inside lists and maps) to dicts, e.g. via `as_dict()`, before calling
`pa.array` for struct-typed fields fixes it. I can open a PR with a regression
test in `arrow_type_compatibility_test.py` / `parquetio_test.py`.
Related: apache/beam#41100 (`ReadFromParquet(as_rows=True)` yields dicts for
struct fields). Together they mean struct columns can't round-trip Parquet ->
transform -> Parquet with `beam.Row`.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [x] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
