[jira] [Commented] (HDFS-14702) Datanode.ReplicaMap memory leak

2019-08-05 Thread He Xiaoqiao (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-14702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900585#comment-16900585
 ] 

He Xiaoqiao commented on HDFS-14702:


[~crh], Thanks for your feedback, just notice that HDFS-8859 try to improve 
memory footprint but not backport. `ReplicaMap` information as following,
||Type||Name||Value||
|ref|entrySet|null|
|int|hashSeed|0|
|int|modCount|1261173816|
|float|loadFactor|0.75|
|int|threshold|6291456|
|int|size|4093978|
|ref|table|java.util.HashMap$Entry[8388608] @ 0x78cb249f0|
|ref|values|java.util.HashMap$Values @ 0x778155350|
|ref|keySet|null|
!datanode.dump.png!

> Datanode.ReplicaMap memory leak
> ---
>
> Key: HDFS-14702
> URL: https://issues.apache.org/jira/browse/HDFS-14702
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.7.1
>Reporter: He Xiaoqiao
>Priority: Major
> Attachments: datanode.dump.png
>
>
> DataNode memory is occupied by ReplicaMaps and cause GC high frequency then 
> write performance degrade.
> It is about 600K block replicas located at DataNode, but when dump heap, 
> there are over 8M items of ReplicaMaps and footprint over 500MB. It seems 
> that memory leak. One more situation, the block w/r ops is very high.
> Do not test HDFS-8859 and no idea if it can solve this issue.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-14702) Datanode.ReplicaMap memory leak

2019-08-05 Thread CR Hota (JIRA) 


[ 
https://issues.apache.org/jira/browse/HDFS-14702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900467#comment-16900467
 ] 

CR Hota commented on HDFS-14702:


[~hexiaoqiao] Thanks for reporting this.

Can you try to backport HDFS-8859 to 2.7.1 installation you have and let us 
know how the heap dump looks like for the data node.

> Datanode.ReplicaMap memory leak
> ---
>
> Key: HDFS-14702
> URL: https://issues.apache.org/jira/browse/HDFS-14702
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.7.1
>Reporter: He Xiaoqiao
>Priority: Major
>
> DataNode memory is occupied by ReplicaMaps and cause GC high frequency then 
> write performance degrade.
> It is about 600K block replicas located at DataNode, but when dump heap, 
> there are over 8M items of ReplicaMaps and footprint over 500MB. It seems 
> that memory leak. One more situation, the block w/r ops is very high.
> Do not test HDFS-8859 and no idea if it can solve this issue.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org