[
https://issues.apache.org/jira/browse/HDDS-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935663#comment-16935663
]
Shashikant Banerjee commented on HDDS-2152:
-------------------------------------------
The issue gets recreated when u try to create/write a key of size 300 MB as in
the test with java heap to set to 256 MB or lower. This issue needs some
discussions on how to avoid buffer copies while doing protobuf conversion as
well as dig in areas of code where actual possible buffer copy is happening
while writes or possible memory leaks.
> Ozone client fails with OOM while writing a large (~300MB) key.
> ---------------------------------------------------------------
>
> Key: HDDS-2152
> URL: https://issues.apache.org/jira/browse/HDDS-2152
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
> Reporter: Aravindan Vijayan
> Assignee: YiSheng Lien
> Priority: Major
> Attachments: largekey.png
>
>
> {code}
> dd if=/dev/zero of=testfile bs=1024 count=307200
> ozone sh key put /vol1/bucket1/key testfile
> {code}
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at
> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at
> java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at
> org.apache.hadoop.hdds.scm.storage.BufferPool.allocateBufferIfNeeded(BufferPool.java:66)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:234)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:129)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:211)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:193)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:96) at
> org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:117)
> at
> org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:55)
> at picocli.CommandLine.execute(CommandLine.java:1173) at
> picocli.CommandLine.access$800(CommandLine.java:141)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
