Siddharth Teotia created ARROW-1533:
---------------------------------------
Summary: As part of buffer transfer (transferTo function), we
should transfer the state for realloc
Key: ARROW-1533
URL: https://issues.apache.org/jira/browse/ARROW-1533
Project: Apache Arrow
Issue Type: Bug
Components: Java - Vectors
Reporter: Siddharth Teotia
Assignee: Siddharth Teotia
We recently encountered a problem when we were trying to add JSON files with
complex schema as datasets.
Initially we started with a Float8Vector with default memory allocation of
(4096 * 8) 32KB.
Went through several iterations of setSafe() to trigger a realloc() from 32KB
to 64KB.
Another round of setSafe() calls to trigger a realloc() from 64KB to 128KB
After that we encountered a BigInt and promoted our vector to UnionVector.
This required us to create a UnionVector with BigIntVector and Float8Vector.
The latter required us to transfer the Float8Vector we were earlier working
with to the Float8Vector inside the Union.
As part of transferTo(), the target Float8Vector got all the ArrowBuf state
(capacity, buffer contents) etc transferred from the source vector.
Later, a realloc was triggered on the Float8Vector inside the UnionVector.
The computation inside realloc() to determine the amount of memory to be
reallocated goes wrong since it makes the decision based on allocateSizeInBytes
-- although this vector was created as part of transfer() from 128KB source
vector, allocateSizeInBytes is still at the initial/default value of 32KB
We end up allocating a 64KB buffer and attempt to copy 128KB over 64KB and seg
fault when invoking setBytes().
There is a wrong assumption in realloc() that allocateSizeInBytes is always
equal to data.capacity(). The particular scenario described above exposes where
this assumption could go wrong.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
