thisisnic commented on issue #30648:
URL: https://github.com/apache/arrow/issues/30648#issuecomment-6402274599
Closing this one out, as Weston answered it above: Arrow and Parquet columns
have to be a single type, and there's no Arrow type that holds "a number or a
list or a dict depending on the row", so `pa.array()` has nothing it can infer
for `[24, 27, [1, 1]]`.
@cdl-altium I think the fastparquet behaviour you're remembering isn't
actually nested type support. For object columns of dicts/lists it just
`json.dumps()` each cell and writes a string column. You can do the same thing
with pyarrow, and there's a JSON extension type now so the column is at least
tagged as JSON rather than being a plain string:
```python
import json
import pyarrow as pa
rows = [{"a": 1, "b": [1, 2]}, {"a": "x", "c": {"d": None}}]
arr = pa.array([json.dumps(r) for r in rows], type=pa.json_(pa.string()))
```
If what you want is the structure kept rather than stringified, that's the
Variant type. Parquet has a Variant logical type and Arrow has a canonical
Variant extension type, and the pyarrow side is tracked in #50131 and #50132,
so those are the ones to follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
