jk-shah opened a new issue, #41098:
URL: https://github.com/apache/beam/issues/41098
### What happened?
`beam.io.ReadFromJson` (and therefore Beam YAML `ReadFromJson`) reads nested
JSON objects and arrays as strings holding their Python repr instead of JSON.
For the line `{"id": 1, "user": {"name": "a"}, "tags": ["x", "y"]}` the element
is `id=1, user="{'name': 'a'}", tags="['x', 'y']"`. The single-quoted repr is
not valid JSON, so downstream steps can't recover the nested values, and the
schema reports the field as `Optional[str]`.
Minimal reproducer:
```python
import json, os, tempfile
import apache_beam as beam
with tempfile.TemporaryDirectory() as d:
path = os.path.join(d, 'nested.jsonl')
with open(path, 'w') as f:
f.write('{"id": 1, "user": {"name": "a"}, "tags": ["x", "y"]}\n')
with beam.Pipeline() as p:
(p
| beam.io.ReadFromJson(path)
| beam.Map(lambda r: print(repr(r.user), json.loads(r.user))))
# prints "{'name': 'a'}" and then fails with json.decoder.JSONDecodeError
```
### Cause
`ReadFromJson` delegates to `apache_beam.dataframe.io.ReadViaPandas('json',
...)`. With the default `objects_as_strings=True`, `ReadViaPandas.expand`
(`sdks/python/apache_beam/dataframe/io.py`) casts every object-dtype column
with `df[col].astype(pd.StringDtype())`. For JSON input, pandas keeps nested
objects and arrays as `dict` / `list` values in object-dtype columns, so the
cast stores `str(dict)`.
Encoding `dict` / `list` values with `json.dumps` before the cast (JSON
format only) keeps the column a string but makes it valid JSON. I have a small
fix with a regression test in `textio_test.JsonTest` and can open a PR.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [x] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
