[
https://issues.apache.org/jira/browse/HDDS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955011#comment-16955011
]
Tsz-wo Sze commented on HDDS-2331:
----------------------------------
>From https://issues.apache.org/jira/secure/attachment/12983374/profiler.png ,
>it seems that Ozone client creates a 16MB byte[] as a buffer to store an 1MB
>data and then uses it to create a ByteString so that the 16MB byte[] is
>referenced in the ByteString. Before HDDS-2169, the proto is *copied* to a
>ByteString so that the 16MB byte[] buffer can be released. HDDS-2169 avoids
>the copying and the buffer remains in the ByteString.
In order to fix the problem, we may either (1) avoid creating a 16MB byte[] in
the first place (for example, we may gradually create 1MB byte[] instead) or
(2) give up on avoiding the buffer copying. It seems (1) is a better solution.
BTW, please feel free to revert HDDS-2169 if it is desirable.
> Client OOME due to buffer retention
> -----------------------------------
>
> Key: HDDS-2331
> URL: https://issues.apache.org/jira/browse/HDDS-2331
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
> Affects Versions: 0.5.0
> Reporter: Attila Doroszlai
> Priority: Critical
> Attachments: profiler.png
>
>
> Freon random key generator exhausts default heap after just few hundred 1MB
> keys. Heap dump on OOME reveals 150+ instances of
> {{ContainerCommandRequestMessage}}, each with 16MB {{byte[]}}.
> Steps to reproduce:
> # Start Ozone cluster with 1 datanode
> # Start Freon (5K keys of size 1MB)
> Result: OOME after a few hundred keys
> {noformat}
> $ cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> $ export HADOOP_OPTS='-XX:+HeapDumpOnOutOfMemoryError'
> $ ozone freon rk --numOfThreads 1 --numOfVolumes 1 --numOfBuckets 1
> --replicationType RATIS --factor ONE --keySize 1048576 --numOfKeys 5120
> --bufferSize 65536
> ...
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid289.hprof ...
> Heap dump file created [1456141975 bytes in 7.760 secs]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
