xichen01 commented on PR #4572: URL: https://github.com/apache/ozone/pull/4572#issuecomment-1539952753
> > In my tests, this does reduce memory usage. > > I found that it is not the PartKeyInfo that consumes the most memory, but the Map.entry where the PartKeyInfo is stored. > > `IntSparseMap` is unnecessarily complicated. Once the TreeMap is built, it won't be updated anymore. Why not simply (1) build a TreeMap, (2) copy the result to a sorted (immutable) list, (3) pass the list around (instead of passing a TreeMap) and (4) use binary search to lookup parts? Thanks for your suggestion, I think we need a `sorted list` that consumes as little memory as possible on itself, `Array` are more appropriate here, because if it is a `List`, each list node will have at least `perv`, `next`, `value` members, and those will still consume a lot of memory with the O(n^2) problem. `IntSparseMap` is actually a wrapper `ArrayList`, and if you want to make it even simpler, one option I have in mind is: We only create `TreeMap` inside `OmMultipartKeyInfo`, and when `copy` and `pass` the `partKeyInfos`, we use `ArrayList`, and in `OmMultipartKeyInfo` internally, we convert `ArrayList` to `TreeMap` to support `Add` and `Get` operations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
