jorisvandenbossche commented on issue #40124:
URL: https://github.com/apache/arrow/issues/40124#issuecomment-1966532178

   The current behaviour is somewhat documented:
   
           schema : Schema, default None
               If not passed, will be inferred from the first row of the
               mapping values.
   
   But I agree this might be surprising, or in any case that it can be useful 
to change that behaviour.
   
   Workaround on the short term is to use `pa.array` to infer the list of dicts 
to a StructArray, which has the desired behaviour, and then this array can be 
converted zero copy to a batch or table:
   
   ```
   In [4]: arr = pa.array([{"a": 1, "b": 2}, {"a": 3, "b": 4, "c": 5}])
   
   In [5]: batch = pa.RecordBatch.from_struct_array(arr)
   
   In [6]: batch
   Out[6]: 
   pyarrow.RecordBatch
   a: int64
   b: int64
   c: int64
   ----
   a: [1,3]
   b: [2,4]
   c: [null,5]
   
   In [7]: batch.to_pandas()
   Out[7]: 
      a  b    c
   0  1  2  NaN
   1  3  4  5.0
   ```
   
   I suppose for larger data, this should actually also be faster, and so we 
should maybe consider using that under the hood as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to