[GitHub] [incubator-ratis] runzhiwang opened a new pull request #76: RATIS-845. Memory leak of RaftServerImpl

Wed, 29 Apr 2020 23:11:21 -0700 


runzhiwang opened a new pull request #76:
URL: https://github.com/apache/incubator-ratis/pull/76


   **What's the problem ?**
   As the image shows, there are 1885 instances of  RaftServerImpl, most of 
them are Closed, and should be GC, but actually not. You can find from the 
image 
    1513 RaftServerImpl were held by 
ManagermentFactory->jxmMBeanServer->HashMap, 372 RaftServerImpl were held by 
Datanode ReportManager Thread -> prometheus -> HashMap. So 1513 RaftServerImpl 
leak in ratis, and 372 leak in ozone. If RaftServerImpl can not GC, there are a 
lot of related resource can not be GC, such as the 
[DirectByteBuffer](https://github.com/apache/incubator-ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java#L150)
  in SegmentRaftLogWorker, which result 1GB memory leak out of heap.
   
   **1. 1885 instances of RaftServerImpl**
   
![image](https://user-images.githubusercontent.com/51938049/80677661-ddb81d80-8aeb-11ea-8664-8a340d19ffb6.png)
   
   **2. 1513 RaftServerImpl were held by 
ManagermentFactory->jxmMBeanServer->HashMap, 372 RaftServerImpl were held by 
Datanode ReportManager Thread -> prometheus -> HashMap**
   
![image](https://user-images.githubusercontent.com/51938049/80677750-09d39e80-8aec-11ea-9da9-4f7b0d31ef9c.png)
   
   **3. 1513 RaftServerImpl were held by 
ManagermentFactory->jxmMBeanServer->HashMap**
   
![image](https://user-images.githubusercontent.com/51938049/80677767-15bf6080-8aec-11ea-8a95-d1f23d52a286.png)
   
   **4. 372 RaftServerImpl were held by Datanode ReportManager Thread -> 
prometheus -> HashMap**
   
![image](https://user-images.githubusercontent.com/51938049/80677785-24a61300-8aec-11ea-8ca4-5c81a44015c5.png)
   
   5. 2038 DirectByteBuffer, and 1885 held by RaftServerImpl
   
![image](https://user-images.githubusercontent.com/51938049/80677796-2a9bf400-8aec-11ea-9734-29a7e62de891.png)
   
   6. 1033 DirectByteBuffer were held by ManagermentFactory, 802 
DirectByteBuffer were held by Datanode ReportManager Thread, total 1885.
   
![image](https://user-images.githubusercontent.com/51938049/80677834-3e475a80-8aec-11ea-933e-76a60dd082f0.png)
   
   7. The reason RaftServerImpl held by 
ManagermentFactory->jxmMBeanServer->HashMap is ratis start 
[JmxReporter](https://github.com/apache/incubator-ratis/blob/master/ratis-metrics/src/main/java/org/apache/ratis/metrics/MetricsReporting.java#L47),
 but does not stop it.
   
   8. The reason RaftServerImpl held by Datanode ReportManager Thread -> 
prometheus -> HashMap is ozone call the ratis function to  
[register](https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java#L189)
 metric in prometheus, but does not unregister it.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to