[
https://issues.apache.org/jira/browse/ARROW-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299424#comment-16299424
]
ASF GitHub Bot commented on ARROW-1943:
---------------------------------------
siddharthteotia commented on issue #1439: ARROW-1943: [JAVA] handle
setInitialCapacity for deeply nested lists
URL: https://github.com/apache/arrow/pull/1439#issuecomment-353239485
I thought more about this and the implemented solution is debatable even
though it fixes the overallocation exception problem.
For example, consider the newly added unit test for schema : LIST (LIST
(INT))
Since each position in the top level vector has 1 or more lists, the number
of offsets in the inner list will always be greater than offsets in its parent.
This implies that there is some factor of increase in capacity as we go down
the tree. In the context of unit test:
The value capacity of outer list is 2 and inner list is 4 because each
position of outer list has 2 inner lists and then we have an int vector with
value capacity 10 comprising of data across all inner lists.
So doing list.setInitialCapacity(2) -> innerList.setInitialCapacity(2) ->
intVector.setInitialCapacity(2 * 5) will require expansion of offset buffer
(and validity) of inner list.
The question really is if there is a reasonable way to increase the
multiplier as we go down the nested lists. The current patch keeps it same
until we get down to scalars and then we directly use a multiplier of 5.
However, this will potentially require re-allocation of internal buffers of
each inner list vector as the user app writes data into deeply nested lists.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Handle setInitialCapacity() for deeply nested lists of lists
> ------------------------------------------------------------
>
> Key: ARROW-1943
> URL: https://issues.apache.org/jira/browse/ARROW-1943
> Project: Apache Arrow
> Issue Type: Bug
> Reporter: Siddharth Teotia
> Assignee: Siddharth Teotia
> Labels: pull-request-available
>
> The current implementation of setInitialCapacity() uses a factor of 5 for
> every level we go into list:
> So if the schema is LIST (LIST (LIST (LIST (LIST (LIST (LIST (BIGINT))))))
> and we start with an initial capacity of 128, we end up throwing
> OversizedAllocationException from the BigIntVector because at every level we
> increased the capacity by 5 and by the time we reached inner scalar that
> actually stores the data, we were well over max size limit per vector (1MB).
> We saw this problem in Dremio when we failed to read deeply nested JSON data.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
