[
https://issues.apache.org/jira/browse/ARROW-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170835#comment-16170835
]
ASF GitHub Bot commented on ARROW-1533:
---------------------------------------
GitHub user siddharthteotia opened a pull request:
https://github.com/apache/arrow/pull/1112
ARROW-1533: [JAVA] realloc should consider the existing buffer capacity for
computing target memory requirement
cc @jacques-n , This is same as https://github.com/apache/arrow/pull/1097
The latter one was closed as I had to rename the branch correctly and use
the correct JIRA number.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/siddharthteotia/arrow ARROW-1533
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/arrow/pull/1112.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1112
----
commit 66620f12f8b0cba359e13ab2e1f6cad21ef4daac
Author: siddharth <[email protected]>
Date: 2017-09-12T21:58:07Z
ARROW-1553: realloc should consider the existing buffer capacity for
computing target memory requirement
commit 75243251d4580febfc094f422fb2780191439a07
Author: siddharth <[email protected]>
Date: 2017-09-17T21:35:38Z
added tests
----
> [JAVA] realloc should consider the existing buffer capacity for computing
> target memory requirement
> ---------------------------------------------------------------------------------------------------
>
> Key: ARROW-1533
> URL: https://issues.apache.org/jira/browse/ARROW-1533
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Vectors
> Reporter: Siddharth Teotia
> Assignee: Siddharth Teotia
> Labels: pull-request-available
>
> We recently encountered a problem when we were trying to add JSON files with
> complex schema as datasets.
> Initially we started with a Float8Vector with default memory allocation of
> (4096 * 8) 32KB.
> Went through several iterations of setSafe() to trigger a realloc() from 32KB
> to 64KB.
> Another round of setSafe() calls to trigger a realloc() from 64KB to 128KB
> After that we encountered a BigInt and promoted our vector to UnionVector.
> This required us to create a UnionVector with BigIntVector and Float8Vector.
> The latter required us to transfer the Float8Vector we were earlier working
> with to the Float8Vector inside the Union.
> As part of transferTo(), the target Float8Vector got all the ArrowBuf state
> (capacity, buffer contents) etc transferred from the source vector.
> Later, a realloc was triggered on the Float8Vector inside the UnionVector.
> The computation inside realloc() to determine the amount of memory to be
> reallocated goes wrong since it makes the decision based on
> allocateSizeInBytes -- although this vector was created as part of transfer()
> from 128KB source vector, allocateSizeInBytes is still at the initial/default
> value of 32KB
> We end up allocating a 64KB buffer and attempt to copy 128KB over 64KB and
> seg fault when invoking setBytes().
> There is a wrong assumption in realloc() that allocateSizeInBytes is always
> equal to data.capacity(). The particular scenario described above exposes
> where this assumption could go wrong.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
