Liya Fan created ARROW-5386:
-------------------------------
Summary: Making the rounding behavior of the buffer capacity
configurable
Key: ARROW-5386
URL: https://issues.apache.org/jira/browse/ARROW-5386
Project: Apache Arrow
Issue Type: Improvement
Components: Java
Reporter: Liya Fan
Assignee: Liya Fan
In our scenario, the following code snippet is frequent in our code base:
int requestSize = ...;
if (requestSize <= allocator.getLimit() - allocator.getAllocatedMemory()) {
ArrowBuf buffer = allocator.buffer(requestSize);
}
However, it often causes OutOfMemoryException, due to Arrow's rounding behavior.
For example, we have only 12 MB memory left, and we request a buffer with size
10 MB. Appearantly, there is sufficient memory to meet the request. However,
the rounding behavior rounds the request size from 10 MB to 16 MB, and there is
no 16 MB memory, so an OutOfMemoryException will be thrown.
We propose two ways to solve this problem:
1. We provide a rounding option as an argument to the BaseAllocator#buf method.
There are two possible values for the rounding option: rounding up and rounding
down. In the above scenario, the rounding down option can solve the problem.
2. We add a method to the allocator:
int getRoundedSize(final int size, BaseAllocator.AllocationRoundingOption
roundingOption)
This method will give the rounding buffer size, given the initial request size.
With this method, the user can freely adjust their request size to avoid OOM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
