[
https://issues.apache.org/jira/browse/HADOOP-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793406#action_12793406
]
Suresh Srinivas commented on HADOOP-6460:
-----------------------------------------
IPC handler uses ByteArrayOutputStream for serializing response. It grows
doubling every time. Any large response results in growth of this buffer,
resulting in large heap usage. This memory is never reclaimed by GC.
On namenode, the following RPC calls have huge impact on the buffer growth:
# publi# BlocksWithLocations NameNodeProtocol.getBlocks(DatanodeInfo datanode,
long size)
# public DatanodeInfo[]
ClientProtocol.getDatanodeReport(FSConstants.DatanodeReportType type)
# public FileStatus[] ClientProtocol.getListing(String src)
Of this, I think getBlocks() is the issue. On production clusters I saw this
heap growing to 167MB. This does not explain buffers growing to the size of
167MB (byte[] grew to as big as 167772160 entries). Given this kind of large
responses, we should relinquish the old buffer and create a new one, when ever
the response buffer grows to a large size and let the old buffer garbage
collected. This is safe, as currently the buffer is used as intermediate
storage for serializing the response.
> Namenode runs of out of memory due to memory leak in ipc Server
> ---------------------------------------------------------------
>
> Key: HADOOP-6460
> URL: https://issues.apache.org/jira/browse/HADOOP-6460
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0, 0.22.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.2, 0.21.0, 0.22.0
>
>
> Namenode heap usage grows disproportional to the number objects supports
> (files, directories and blocks). Based on heap dump analysis, this is due to
> large growth in ByteArrayOutputStream allocated in
> o.a.h.ipc.Server.Handler.run().
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
