xichen01 commented on PR #4572: URL: https://github.com/apache/ozone/pull/4572#issuecomment-1516532379
> @xichen01 , thanks for working on this! > > * Because of the underlying O(n^2) problem, this memory improvement is minor since it does not reduce the size or the number of the `PartKeyInfo` objects (~1.5 KB each). > * For 10,000 parts, there are 50,005,000 `PartKeyInfo` objects as you mentioned, and the size is ~71GB. The improvement here is 1.7GB (=2.17GB - 491MB), which is around 2.3% (=1.7/(71 + 2.17)). > * Once the O(n^2) problem has been fixed, the `TreeMap` problem in this pull request will disappear. > > In the change here, you have avoided copying a `TreeMap` in a `OmMultipartKeyInfo` constructor. How about keep the `TreeMap` and avoid the copying, but not adding `IntSparseMap`? It is also a solution, where I have tried to use change this https://github.com/apache/ozone/blob/cffa3864b974499078fbdb37506ed992319b09c5/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java#L149-L150 to ```java multipartKeyInfo = omMetadataManager.getMultipartInfoTable() .getReadCopy(multipartKey); ``` But there is a `ConcurrentModificationException` in the `DoubleBuffer`, so we may also need to do some state synchronization. But I'm not sure if this is a safe solution. Here are some references to the DoubleBuffer's `ConcurrentModificationException` https://issues.apache.org/jira/browse/HDDS-2344 https://issues.apache.org/jira/browse/HDDS-3623 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
