[ https://issues.apache.org/jira/browse/PARQUET-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771730#comment-16771730 ]
Gabor Szadovszky commented on PARQUET-1533: ------------------------------------------- Thanks a lot, [~sha...@uber.com] for creating this JIRA. I haven't realized during the review that we use such large default buffer sizes. [~liupengcheng], is there a reason behind the default 64M buffer size? (Parquet uses compression on pages which are usually ~1M size.) The original implementation did not use any default size but simply initialized the buffers with 0 and let them grow. Would you like to handle this issue? > TestSnappy() throws OOM exception with Parquet-1485 change > ----------------------------------------------------------- > > Key: PARQUET-1533 > URL: https://issues.apache.org/jira/browse/PARQUET-1533 > Project: Parquet > Issue Type: Bug > Components: parquet-mr > Affects Versions: 1.11.0 > Environment: Mac OS 10.14.1 > Reporter: Xinli Shang > Priority: Minor > > Parquet-1485 initialize the buffer size(inputBuffer and outputBuffer) from 0 > to 128M in total. This cause the unit test TestSnappy() failed with OOM > exception. This is on my Mac laptop. > To solve the unit test failure, we can increase the size of -Xmx from 512m to > 1024m like below. However, we need to evaluate whether or not the increase of > the initial size of direct memory usage for inputBuffer and outputBuffer will > cause real Parquet application OOM or not, if that application is not with > big enough -Xmx size. > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-surefire-plugin</artifactId> > ... > <argLine>-Xmx1014m</argLine> > ... > For details of the exception, the pull > request(https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b) > has the detail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)