[
https://issues.apache.org/jira/browse/PARQUET-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771730#comment-16771730
]
Gabor Szadovszky commented on PARQUET-1533:
-------------------------------------------
Thanks a lot, [[email protected]] for creating this JIRA. I haven't realized
during the review that we use such large default buffer sizes.
[~liupengcheng], is there a reason behind the default 64M buffer size? (Parquet
uses compression on pages which are usually ~1M size.) The original
implementation did not use any default size but simply initialized the buffers
with 0 and let them grow. Would you like to handle this issue?
> TestSnappy() throws OOM exception with Parquet-1485 change
> -----------------------------------------------------------
>
> Key: PARQUET-1533
> URL: https://issues.apache.org/jira/browse/PARQUET-1533
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.11.0
> Environment: Mac OS 10.14.1
> Reporter: Xinli Shang
> Priority: Minor
>
> Parquet-1485 initialize the buffer size(inputBuffer and outputBuffer) from 0
> to 128M in total. This cause the unit test TestSnappy() failed with OOM
> exception. This is on my Mac laptop.
> To solve the unit test failure, we can increase the size of -Xmx from 512m to
> 1024m like below. However, we need to evaluate whether or not the increase of
> the initial size of direct memory usage for inputBuffer and outputBuffer will
> cause real Parquet application OOM or not, if that application is not with
> big enough -Xmx size.
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> ...
> <argLine>-Xmx1014m</argLine>
> ...
> For details of the exception, the pull
> request(https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b)
> has the detail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
