jorisvandenbossche commented on issue #40124:
URL: https://github.com/apache/arrow/issues/40124#issuecomment-1966532178
The current behaviour is somewhat documented:
schema : Schema, default None
If not passed, will be inferred from the first row of the
mapping values.
But I agree this might be surprising, or in any case that it can be useful
to change that behaviour.
Workaround on the short term is to use `pa.array` to infer the list of dicts
to a StructArray, which has the desired behaviour, and then this array can be
converted zero copy to a batch or table:
```
In [4]: arr = pa.array([{"a": 1, "b": 2}, {"a": 3, "b": 4, "c": 5}])
In [5]: batch = pa.RecordBatch.from_struct_array(arr)
In [6]: batch
Out[6]:
pyarrow.RecordBatch
a: int64
b: int64
c: int64
----
a: [1,3]
b: [2,4]
c: [null,5]
In [7]: batch.to_pandas()
Out[7]:
a b c
0 1 2 NaN
1 3 4 5.0
```
I suppose for larger data, this should actually also be faster, and so we
should maybe consider using that under the hood as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]