[
https://issues.apache.org/jira/browse/ARROW-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fangzhou updated ARROW-198:
---------------------------
Description:
When executing vector component junit-test class TestValueVector,exception can
be seen.
Exception:
java.lang.Exception: Unexpected exception,
expected<org.apache.arrow.vector.util.OversizedAllocationException> but
was<org.apache.arrow.memory.OutOfMemoryException>
where the issue comes from:
public void testFixedVectorReallocation() {
....
try {
vector.allocateNew(expectedValueCapacity); <-- where the exception
comes from
....
analysis:
allocateNew will invoke classes from io.netty.buffer from memory component and
finally invoke netty third party package.I assume there is no issue when
invoking memory part.Then I trace that the issue is caused by oversizing
value of initialCapacity setting when invoking AbstractByteBufAllocator class
in third party netty-buffer-4.0.27.final.jar. I create a test case which can
reproduces the issue. value of parameter initialCapacity and maxCapacity are
obtained from debugging procedure
public void testAllocator(){
int initialCapacity=2147483644;
int maxCapacity=2147483647;
ByteBuf buf =
UnpooledByteBufAllocator.DEFAULT.directBuffer(initialCapacity, maxCapacity);
}
And I checked the rooted affected class java.nio.Bits:
static void reserveMemory(long size, int cap) {
.....
if (totalCapacity + cap > maxMemory)
throw new OutOfMemoryError("Direct buffer memory");
.....
maxMemory can only be set to 1836056576 at most,but cap(initialCapacity) was
2147483644 at this moment.
was:
When executing vector component junit-test class TestValueVector,exception can
be seen.
Exception:
java.lang.Exception: Unexpected exception,
expected<org.apache.arrow.vector.util.OversizedAllocationException> but
was<org.apache.arrow.memory.OutOfMemoryException>
where the issue comes from:
public void testFixedVectorReallocation() {
....
try {
vector.allocateNew(expectedValueCapacity); <-- where the exception
comes from
....
analysis:
allocateNew will invoke classes from io.netty.buffer from memory component and
finally invoke netty third party package.I assume there is no issue when
invoking memory part.Then I trace that the issue is caused by oversizing
value of initialCapacity setting when invoking AbstractByteBufAllocator class
in third party netty-buffer-4.0.27.final.jar. I create a test case which can
reproduces the issue. value of parameter initialCapacity and maxCapacity are
obtained from debugging procedure
public void testAllocator(){
int initialCapacity=2147483644;
int maxCapacity=2147483647;
ByteBuf buf =
UnpooledByteBufAllocator.DEFAULT.directBuffer(initialCapacity, maxCapacity);
}
And I checked the rooted affected class java.nio.Bits:
static void reserveMemory(long size, int cap) {
.....
if (totalCapacity + cap > maxMemory)
throw new OutOfMemoryError("Direct buffer memory");
.....
maxMemory can only be set to 1836056576 at most,but initialCapacity was
2147483644 at this moment.
> OutOfMemoryError for vector test case
> -------------------------------------
>
> Key: ARROW-198
> URL: https://issues.apache.org/jira/browse/ARROW-198
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java - Memory, Java - Vectors
> Reporter: Fangzhou
>
> When executing vector component junit-test class TestValueVector,exception
> can be seen.
> Exception:
> java.lang.Exception: Unexpected exception,
> expected<org.apache.arrow.vector.util.OversizedAllocationException> but
> was<org.apache.arrow.memory.OutOfMemoryException>
> where the issue comes from:
> public void testFixedVectorReallocation() {
> ....
> try {
> vector.allocateNew(expectedValueCapacity); <-- where the exception
> comes from
> ....
> analysis:
> allocateNew will invoke classes from io.netty.buffer from memory component
> and finally invoke netty third party package.I assume there is no issue when
> invoking memory part.Then I trace that the issue is caused by oversizing
> value of initialCapacity setting when invoking AbstractByteBufAllocator class
> in third party netty-buffer-4.0.27.final.jar. I create a test case which can
> reproduces the issue. value of parameter initialCapacity and maxCapacity are
> obtained from debugging procedure
> public void testAllocator(){
> int initialCapacity=2147483644;
> int maxCapacity=2147483647;
> ByteBuf buf =
> UnpooledByteBufAllocator.DEFAULT.directBuffer(initialCapacity, maxCapacity);
> }
> And I checked the rooted affected class java.nio.Bits:
> static void reserveMemory(long size, int cap) {
> .....
> if (totalCapacity + cap > maxMemory)
> throw new OutOfMemoryError("Direct buffer memory");
> .....
> maxMemory can only be set to 1836056576 at most,but cap(initialCapacity) was
> 2147483644 at this moment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
