[
https://issues.apache.org/jira/browse/ARROW-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787352#comment-16787352
]
Wes McKinney commented on ARROW-4757:
-------------------------------------
Can you write a stream (multiple IPC messages) rather than a single one? I
won't be able to look at this for 0.13
> Nested chunked array support
> ----------------------------
>
> Key: ARROW-4757
> URL: https://issues.apache.org/jira/browse/ARROW-4757
> Project: Apache Arrow
> Issue Type: Improvement
> Reporter: Philipp Moritz
> Priority: Major
> Fix For: 0.14.0
>
>
> Dear all,
> I'm currently trying to lift the 2GB limit on the python serialization. For
> this, I implemented a chunked union builder to split the array into smaller
> arrays.
> However, some of the children of the union array can be ListArrays, which can
> themselves contain UnionArrays which can contain ListArrays etc. I'm at a bit
> of a loss how to handle this. In principle I'd like to chunk the children
> too. However, currently UnionArrays can only have children of type Array, and
> there is no way to treat a chunked array (which is a vector of Arrays) as an
> Array to store it as a child of a UnionArray. Any ideas how to best support
> this use case?
> -- Philipp.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
