Anurag-S1ngh opened a new pull request, #41335:
URL: https://github.com/apache/beam/pull/41335
## Summary
`ReadFromJson` (and Beam YAML `ReadFromJson`) read nested JSON objects and
arrays as strings holding their Python repr instead of JSON. For `{"id": 1,
"user": {"name": "a"}, "tags": ["x", "y"]}`, the element was `id=1,
user="{'name': 'a'}", tags="['x', 'y']"`. The single-quoted repr is not valid
JSON, so downstream steps cannot recover the nested values, and the schema
reports the field as `Optional[str]`.
With the default `objects_as_strings=True`, `ReadViaPandas.expand` cast
every object-dtype column with `astype(pd.StringDtype())`. For JSON input,
pandas keeps nested objects and arrays as `dict`/`list` values in object-dtype
columns, so the cast stored `str(dict)`.
The fix encodes `dict`/`list` values with `json.dumps` before the cast, for
the `json` format only, which keeps the column a string but yields valid JSON.
Fixes #41098
## Testing
- New: `textio_test.py::JsonTest::test_nested_values_are_valid_json`
- `pytest sdks/python/apache_beam/io/textio_test.py::JsonTest` (4 passed)
- Issue reproducer no longer prints `{'name': 'a'}`; prints `{"name": "a"}`
and `["x", "y"]` (both parse with `json.loads`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
