[ 
https://issues.apache.org/jira/browse/ARROW-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209157#comment-16209157
 ] 

ASF GitHub Bot commented on ARROW-1584:
---------------------------------------

GitHub user buhrmann opened an issue:

    https://github.com/apache/arrow/issues/1208

    Serializing list arrays containing only empty lists: 
(ArrowNotImplementedError: Unable to convert type: null)

    Hi,
    Not sure if this is related or the same as ARROW-1584, but I can't seem to 
find a way to handle arrays of lists which occasionally consist of empty lists 
only.
    
    To reproduce:
    
    ```
    na = [] # None, [""]
    
    arrays = {
        'c1': pa.array([["test"], na, na], type=pa.list_(pa.string())),
        'c2': pa.array([na, na, na], type=pa.list_(pa.string())),
    }
    
    rb = pa.RecordBatch.from_arrays(list(arrays.values()), list(arrays.keys()))
    df = rb.to_pandas()
    
    pa.serialize_pandas(df)
    # > ArrowNotImplementedError: Unable to convert type: null
    
    tbl = pa.Table.from_pandas(df)
    sink = pa.BufferOutputStream()
    writer = pa.RecordBatchFileWriter(sink, tbl.schema)
    writer.write_table(tbl)
    # > ArrowNotImplementedError: Unable to convert type: null
    ```
    
    In my use case I'm processing data in batches where individual fields 
contain lists of strings. Some of the batches may, however, contain empty lists 
only. And there doesn't seem to be any representation in Arrow at the moment to 
deal with this situation. 
    
    Also, since I'm serializing the batches into a single file/stream, their 
schemas need to be consistent, which is why I tried explicitly specifying the 
type of the array as list_(string). The only workaround I've found is to 
replace empty lists with [""], but that implies lots of unnecessary glue code 
on the client side. Is there a better workaround until this is fixed in an 
official conda release?
    
    Many thanks,
    Thomas

----

----


> [PYTHON] serialize_pandas on empty dataframe
> --------------------------------------------
>
>                 Key: ARROW-1584
>                 URL: https://issues.apache.org/jira/browse/ARROW-1584
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.7.0
>            Reporter: Tom Augspurger
>            Assignee: Wes McKinney
>            Priority: Minor
>              Labels: pull-request-available, python, serialization
>             Fix For: 0.8.0
>
>
> This code
> {code:python}
> import pandas as pd
> import pyarrow as pa
> pa.serialize_pandas(pd.DataFrame())
> {code}
> Raises
> {code}
> ---------------------------------------------------------------------------
> ArrowNotImplementedError                  Traceback (most recent call last)
> <ipython-input-71-ad21add45f0d> in <module>()
> ----> 1 pa.serialize_pandas(pd.DataFrame())
> ~/Envs/dask-dev/lib/python3.6/site-packages/pyarrow/ipc.py in 
> serialize_pandas(df)
>     158     sink = pa.BufferOutputStream()
>     159     writer = pa.RecordBatchStreamWriter(sink, batch.schema)
> --> 160     writer.write_batch(batch)
>     161     writer.close()
>     162     return sink.get_result()
> pyarrow/ipc.pxi in pyarrow.lib._RecordBatchWriter.write_batch 
> (/Users/travis/build/apache/arrow-dist/arrow/python/build/temp.macosx-10.6-intel-3.6/lib.cxx:59238)()
> pyarrow/error.pxi in pyarrow.lib.check_status 
> (/Users/travis/build/apache/arrow-dist/arrow/python/build/temp.macosx-10.6-intel-3.6/lib.cxx:8113)()
> ArrowNotImplementedError: Unable to convert type: null
> {code}
> Presumably {{pa.deserialize_pandas}} will need a fix as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to