[
https://issues.apache.org/jira/browse/HDFS-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316626#comment-16316626
]
Robert Joseph Evans commented on HDFS-12984:
--------------------------------------------
[~ajayydv],
I also ran into issues trying to reproduce this in some environments.
Specifically I could never make it happen on my MBP and I don't know why. But
if you look at the code inside the BlockPoolSlice
https://github.com/apache/hadoop/blob/01f3f2167ec20b52a18bc2cf250fb4229cfd2c14/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java#L165-L173
If an instance of this is ever created it can never be collected. I am not
sure why BlockPoolSlice instances are created some times by a MiniDFSCluster
and not others. I am not that familiar with the internals of the DataNode to
say off the top of my head. Glad to see you going in the right direction, and
I agree that removing everything from the ShutdownHooksManager is far from
ideal, but I didn't see this happening, at least not with 2.7.5 and 2.6.2.
> BlockPoolSlice can leak in a mini dfs cluster
> ---------------------------------------------
>
> Key: HDFS-12984
> URL: https://issues.apache.org/jira/browse/HDFS-12984
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.7.5
> Reporter: Robert Joseph Evans
> Assignee: Ajay Kumar
> Attachments: Screen Shot 2018-01-05 at 4.38.06 PM.png, Screen Shot
> 2018-01-05 at 5.26.54 PM.png, Screen Shot 2018-01-05 at 5.31.52 PM.png
>
>
> When running some unit tests for storm we found that we would occasionally
> get out of memory errors on the HDFS integration tests.
> When I got a heap dump I found that the ShutdownHookManager was full of
> BlockPoolSlice$1 instances. Which hold a reference to the BlockPoolSlice
> which then in turn holds a reference to the DataNode etc....
> It looks like when shutdown is called on the BlockPoolSlice there is no way
> to remove the shut down hook in because no reference to it is saved.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
