Mukund Thakur created HADOOP-18296:
--------------------------------------
Summary: Memory fragmentation in ChecksumFileSystem Vectored IO
implementation.
Key: HADOOP-18296
URL: https://issues.apache.org/jira/browse/HADOOP-18296
Project: Hadoop Common
Issue Type: Sub-task
Components: common
Affects Versions: 3.4.0
Reporter: Mukund Thakur
As we have implemented merging of ranges in the ChecksumFSInputChecker
implementation of vectored IO api, it can lead to memory fragmentation. Let me
explain by example.
Suppose client requests for 3 ranges.
0-500, 700-1000 and 1200-1500.
Now because of merging, all the above ranges will get merged into one and we
will allocate a big byte buffer of 0-1500 size but return sliced byte buffers
for the desired ranges.
Now once the client is done reading all the ranges, it will only be able to
free the memory for requested ranges and memory of the gaps will never be
released for eg here (500-700 and 1000-1200).
Note this only happens for direct byte buffers.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
