xichen01 opened a new pull request, #4572: URL: https://github.com/apache/ozone/pull/4572
## What changes were proposed in this pull request? Optimize the memory usage of `multiupload`. The current `multiupload` will use `TreeMap` to record the information of uploaded "part key", when there are more "part key", `TreeMap` will consume a lot of memory, this PR has a custom `IntSparseMap` class, which is used to replace `TreeMap`. compare use `IntSparseMap` with `TreeMap` | part number | this PR (IntSparseMap) | before this PR (TreeMap) | | ------ | ------ | ------ | | 5000 | 184MB | 491MB | | 10000 | 491MB | 2.17GB | ---- When `multiupload` the main memory consumption is `TreeMap#Entry`, because the current Ozone uses `OmMultipartKeyInfo#partKeyInfoList(TreeMap<Integer, PartKeyInfo>)` to store `PartkeyInfo`, each partition will store itself and the previous all uploaded partition information, which causes each "part key" upload to create a `TreeMap`. For example, for the 1000th "part key", a `TreeMap` with 1000 nodes may be created and for the 10000th "part key", a `TreeMap` with 10000 nodes may be created. `IntSparseMap` internally uses an array to store the partkey information, because each node of the array itself does not occupy memory, so it finally achieves the purpose of saving memory ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-8427 ## How was this patch tested? unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
