[GitHub] [ozone] devmadhuu opened a new pull request, #3974: HDDS-7500. Ozone recon server possible memory leak.

[GitBox]

devmadhuu opened a new pull request, #3974:
URL: https://github.com/apache/ozone/pull/3974

   Issue:
   we are keep on iterating that iterator using while loop and 
"handlePutDirEvent" method keep pushing the keys and value objects (NSSummary) 
in `nsSummaryMap` and that hashmap is retaining lot of objects worth of 16 GB 
and then eventually goes OOM. At the end of while loop down the code in this 
method, we push the data to NSSummaryTable in recon SQL DB.
   
   <img width="695" alt="Screenshot 2022-11-17 at 11 21 28 AM" 
src="https://user-images.githubusercontent.com/20607992/202446439-89936e40-2fe3-432b-abfd-51ade5637619.png";>
   <img width="1689" alt="image" 
src="https://user-images.githubusercontent.com/20607992/202446512-1af61be4-8b81-4746-b655-dccd1d8d7292.png";>
   
   
   Solution:
   As per heap dump provided, further analysed recon code which runs NSSummary 
task, the code which is adding these objects "NSSummary" in hashmap. 
"NSSummary" object is used for storing metadata summary of dir/keys using few 
data structures and all internal data structures are using primitive types for 
storing this metadata which are already optimized. As per heapdump analysis, it 
is being observed that this each metadata object is occupying ~7KB on an 
average if we take out the average of all ~25 million records., So now we can 
think of a simpler solution to keep tab on Recon memory stepping upto max of 1 
GB during this NSSummary task operation, we can do below simple math:
   
   1 GB = 1024 * 1024 KB
   Number of elements to load in HashMap before flushing to DB = (1024 * 1024) 
/ 7 = ~150K entries,
   
   After 150K entries in HashMap, we can flush to DB and then clear out the 
hashmap to reuse and memory will be freed after GC.
   
   https://issues.apache.org/jira/browse/HDDS-7500
   
   and you need to set the title of the pull request which starts with
   the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
   
   Please replace this section with the link to the Apache JIRA)
   
   ## How was this patch tested?
   
   Tested Manually with test cases by generating keys and directories using 
freon and made hit the code.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org