[
https://issues.apache.org/jira/browse/ARROW-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443963#comment-16443963
]
Krisztian Szucs commented on ARROW-2476:
----------------------------------------
I wasn't thinking of the datatype the array holds, just the maximum length of
the arrays.
In the sidenote ment the number of
[offsets|https://github.com/apache/arrow/blob/master/cpp/src/arrow/array.cc#L204]
(just can't find the right impl line).
According to [array lengths
section|https://github.com/apache/arrow/blob/master/format/Layout.md#array-lengths]
I'd think that each array can have at most (2^31 - 1) elements.
Why does it apply to the [List<List<byte>>
example|https://github.com/apache/arrow/blob/master/format/Layout.md#example-layout-listlistbyte]
(including the top array and the nested ones) but doesn't to an array of
primitive type like `array<int32>`?
> [Python/Question] Maximum length of an Array created from ndarray
> -----------------------------------------------------------------
>
> Key: ARROW-2476
> URL: https://issues.apache.org/jira/browse/ARROW-2476
> Project: Apache Arrow
> Issue Type: Improvement
> Reporter: Krisztian Szucs
> Priority: Minor
>
> So the format
> [describes|https://github.com/apache/arrow/blob/master/format/Layout.md#array-lengths]
> that an array max length is 2^31 - 1, however the following python snippet
> creates a 2**32 length arrow array:
> {code:python}
> a = np.ones((2**32,), dtype='int8')
> A = pa.Array.from_pandas(a)
> type(A)
> {code}
> {code}pyarrow.lib.Int8Array{code}
> Based the layout specification I'd expect a ChunkedArray of three Int8Array's
> with lengths:
> [2^31 - 1, 2^31 - 1, 2] or should raise an exception?
> If it's the expectation is there any documentation for it?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
