[
https://issues.apache.org/jira/browse/ARROW-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170963#comment-16170963
]
ASF GitHub Bot commented on ARROW-1533:
---------------------------------------
Github user siddharthteotia commented on the issue:
https://github.com/apache/arrow/pull/1112
@wesm , For easy self-tracking, I typically have the same local branch name
as the one in my fork that I used to create the PR.
For this JIRA (ARROW-1533), I incorrectly used JIRA# 1553 almost everywhere
-- branch name, commit message etc.
I didn't realize this until I had to actually work on ARROW-1553. So
renamed the local branch and did a force update on my fork to get everything
proper for ARROW-1533.
> [JAVA] realloc should consider the existing buffer capacity for computing
> target memory requirement
> ---------------------------------------------------------------------------------------------------
>
> Key: ARROW-1533
> URL: https://issues.apache.org/jira/browse/ARROW-1533
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Vectors
> Reporter: Siddharth Teotia
> Assignee: Siddharth Teotia
> Labels: pull-request-available
>
> We recently encountered a problem when we were trying to add JSON files with
> complex schema as datasets.
> Initially we started with a Float8Vector with default memory allocation of
> (4096 * 8) 32KB.
> Went through several iterations of setSafe() to trigger a realloc() from 32KB
> to 64KB.
> Another round of setSafe() calls to trigger a realloc() from 64KB to 128KB
> After that we encountered a BigInt and promoted our vector to UnionVector.
> This required us to create a UnionVector with BigIntVector and Float8Vector.
> The latter required us to transfer the Float8Vector we were earlier working
> with to the Float8Vector inside the Union.
> As part of transferTo(), the target Float8Vector got all the ArrowBuf state
> (capacity, buffer contents) etc transferred from the source vector.
> Later, a realloc was triggered on the Float8Vector inside the UnionVector.
> The computation inside realloc() to determine the amount of memory to be
> reallocated goes wrong since it makes the decision based on
> allocateSizeInBytes -- although this vector was created as part of transfer()
> from 128KB source vector, allocateSizeInBytes is still at the initial/default
> value of 32KB
> We end up allocating a 64KB buffer and attempt to copy 128KB over 64KB and
> seg fault when invoking setBytes().
> There is a wrong assumption in realloc() that allocateSizeInBytes is always
> equal to data.capacity(). The particular scenario described above exposes
> where this assumption could go wrong.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
