[
https://issues.apache.org/jira/browse/HDDS-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Supratim Deka updated HDDS-1082:
--------------------------------
Attachment: HDDS-1082.000.patch
Status: Patch Available (was: Open)
Fix summary:
BlockInputStream.read() now marks buffers to null when serving the final
instalment of data in the block.
getPos() required an adjustment. Because getPos() can be called even after all
the data in the stream has been read. The current behaviour of getPos() is to
use the buffers.get(bufferIndex).position() to determine the next read position
in the stream. The patch changes this behaviour because buffers are released
once all data has been read.
Testing:
Acceptance tests basic, ozonefs and s3
I can give a walkthrough of the change, pl ping me.
> OutOfMemoryError while reading key of size 100GB
> ------------------------------------------------
>
> Key: HDDS-1082
> URL: https://issues.apache.org/jira/browse/HDDS-1082
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Nilotpal Nandi
> Assignee: Supratim Deka
> Priority: Blocker
> Fix For: 0.4.0
>
> Attachments: HDDS-1082.000.patch, HDDS-1082.000.patch
>
>
> steps taken :
> --------------------
> # put key with size 100GB
> # Tried to read back the key.
> error thrown:
> ------------------------------
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to /tmp/heapdump.bin ...
> Heap dump file created [3883178021 bytes in 10.667 secs]
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.ratis.thirdparty.com.google.protobuf.ByteString.toByteArray(ByteString.java:643)
> at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:188)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:130)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> at
> org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:49)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> at
> org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> at
> org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> at picocli.CommandLine.execute(CommandLine.java:919)
> at picocli.CommandLine.access$700(CommandLine.java:104)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> at
> picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:83){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
