[ 
https://issues.apache.org/jira/browse/ARROW-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395321#comment-16395321
 ] 

Chris Ellison commented on ARROW-2227:
--------------------------------------

Just wanted to mention, in case it was missed, but this example isn't a single 
large 2 GiB string. Each row in the data frame is a single byte. So it is a 
large array of small bytes. 

Is 0.10.0 far away? Out of the box, using pyarrow for "big data" isn't really 
possible (assuming you have string data) until this is fixed. My hack fix was 
to make changes to pandas_compat.py:dataframe_to_arrays() so that it accepted a 
dictionary mapping column names to chunk sizes, and then I manually (and 
somewhat crudely) create a chunked_array. This is then passed to 
Table.from_arrays (mimicking what appears in table.pxi).

> [Python] Table.from_pandas does not create chunked_arrays.
> ----------------------------------------------------------
>
>                 Key: ARROW-2227
>                 URL: https://issues.apache.org/jira/browse/ARROW-2227
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.8.0
>            Reporter: Chris Ellison
>            Assignee: Wes McKinney
>            Priority: Major
>             Fix For: 0.10.0
>
>
> When creating a large enough array, pyarrow raises an exception:
> {code:java}
> import numpy as np
> import pandas as pd
> import pyarrow as pa
> x = list('1' * 2**31)
> y = pd.DataFrame({'x': x})
> t = pa.Table.from_pandas(y)
> # ArrowInvalid: BinaryArrow cannot contain more than 2147483646 bytes, have 
> 2147483647{code}
> The array should be chunked for the user. As is, data frames with >2 GiB in 
> binary data will struggle to get into arrow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to