[
https://issues.apache.org/jira/browse/ARROW-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liya Fan updated ARROW-5386:
----------------------------
Description:
In our scenario, the following code snippet is frequent in our code base:
int requestSize = ...;
if (requestSize <= allocator.getLimit() - allocator.getAllocatedMemory())
{ ArrowBuf buffer = allocator.buffer(requestSize); }
However, it often causes OutOfMemoryException, due to Arrow's rounding behavior.
For example, we have only 12 MB memory left, and we request a buffer with size
10 MB. Appearantly, there is sufficient memory to meet the request. However,
the rounding behavior rounds the request size from 10 MB to 16 MB, and there is
no 16 MB memory, so an OutOfMemoryException will be thrown.
We propose two ways to solve this problem:
1. We provide a rounding option as an argument to the BaseAllocator#buffer
method. There are two possible values for the rounding option: rounding up and
rounding down. In the above scenario, the rounding down option can solve the
problem.
2. We add a method to the allocator:
int getRoundedSize(final int size, BaseAllocator.AllocationRoundingOption
roundingOption)
This method will give the rounded buffer size, given the initial request size.
With this method, the user can freely adjust their request size to avoid OOM.
To make it more convenient to use Arrow, we think both solutions should be
implemented.
was:
In our scenario, the following code snippet is frequent in our code base:
int requestSize = ...;
if (requestSize <= allocator.getLimit() - allocator.getAllocatedMemory())
{ ArrowBuf buffer = allocator.buffer(requestSize); }
However, it often causes OutOfMemoryException, due to Arrow's rounding behavior.
For example, we have only 12 MB memory left, and we request a buffer with size
10 MB. Appearantly, there is sufficient memory to meet the request. However,
the rounding behavior rounds the request size from 10 MB to 16 MB, and there is
no 16 MB memory, so an OutOfMemoryException will be thrown.
We propose two ways to solve this problem:
1. We provide a rounding option as an argument to the BaseAllocator#buf method.
There are two possible values for the rounding option: rounding up and rounding
down. In the above scenario, the rounding down option can solve the problem.
2. We add a method to the allocator:
int getRoundedSize(final int size, BaseAllocator.AllocationRoundingOption
roundingOption)
This method will give the rounding buffer size, given the initial request size.
With this method, the user can freely adjust their request size to avoid OOM.
To make it more convenient to use Arrow, we think both solutions can be
implemented.
> Making the rounding behavior of the buffer capacity configurable
> ----------------------------------------------------------------
>
> Key: ARROW-5386
> URL: https://issues.apache.org/jira/browse/ARROW-5386
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Java
> Reporter: Liya Fan
> Assignee: Liya Fan
> Priority: Major
>
> In our scenario, the following code snippet is frequent in our code base:
> int requestSize = ...;
> if (requestSize <= allocator.getLimit() - allocator.getAllocatedMemory())
> { ArrowBuf buffer = allocator.buffer(requestSize); }
> However, it often causes OutOfMemoryException, due to Arrow's rounding
> behavior.
> For example, we have only 12 MB memory left, and we request a buffer with
> size 10 MB. Appearantly, there is sufficient memory to meet the request.
> However, the rounding behavior rounds the request size from 10 MB to 16 MB,
> and there is no 16 MB memory, so an OutOfMemoryException will be thrown.
> We propose two ways to solve this problem:
> 1. We provide a rounding option as an argument to the BaseAllocator#buffer
> method. There are two possible values for the rounding option: rounding up
> and rounding down. In the above scenario, the rounding down option can solve
> the problem.
> 2. We add a method to the allocator:
> int getRoundedSize(final int size, BaseAllocator.AllocationRoundingOption
> roundingOption)
> This method will give the rounded buffer size, given the initial request
> size. With this method, the user can freely adjust their request size to
> avoid OOM.
> To make it more convenient to use Arrow, we think both solutions should be
> implemented.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
