[
https://issues.apache.org/jira/browse/ARROW-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169427#comment-16169427
]
ASF GitHub Bot commented on ARROW-1533:
---------------------------------------
Github user siddharthteotia commented on the issue:
https://github.com/apache/arrow/pull/1097
@jacques-n , added tests for fixed-width vector, variable width vector and
bit vector. To further test correctness, I verified that without the code
changes, unit tests fail with expected segmentation fault in reAlloc().
At this point, I don't think we need tests for other vector types for the
problem this patch is fixing.
However, I have two test related patches in pipeline
(1) tests for mapvector -- we don't have any unit tests for MapVector
(2) splitAndTransfer for all vector types -- we recently added for bit
vector, union vector, list vector. Need to cover other types too.
I will ensure that those patches cover tests for this scenario as well.
> [JAVA] realloc should consider the existing buffer capacity for computing
> target memory requirement
> ---------------------------------------------------------------------------------------------------
>
> Key: ARROW-1533
> URL: https://issues.apache.org/jira/browse/ARROW-1533
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Vectors
> Reporter: Siddharth Teotia
> Assignee: Siddharth Teotia
> Labels: pull-request-available
>
> We recently encountered a problem when we were trying to add JSON files with
> complex schema as datasets.
> Initially we started with a Float8Vector with default memory allocation of
> (4096 * 8) 32KB.
> Went through several iterations of setSafe() to trigger a realloc() from 32KB
> to 64KB.
> Another round of setSafe() calls to trigger a realloc() from 64KB to 128KB
> After that we encountered a BigInt and promoted our vector to UnionVector.
> This required us to create a UnionVector with BigIntVector and Float8Vector.
> The latter required us to transfer the Float8Vector we were earlier working
> with to the Float8Vector inside the Union.
> As part of transferTo(), the target Float8Vector got all the ArrowBuf state
> (capacity, buffer contents) etc transferred from the source vector.
> Later, a realloc was triggered on the Float8Vector inside the UnionVector.
> The computation inside realloc() to determine the amount of memory to be
> reallocated goes wrong since it makes the decision based on
> allocateSizeInBytes -- although this vector was created as part of transfer()
> from 128KB source vector, allocateSizeInBytes is still at the initial/default
> value of 32KB
> We end up allocating a 64KB buffer and attempt to copy 128KB over 64KB and
> seg fault when invoking setBytes().
> There is a wrong assumption in realloc() that allocateSizeInBytes is always
> equal to data.capacity(). The particular scenario described above exposes
> where this assumption could go wrong.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
