xichen01 commented on code in PR #4572:
URL: https://github.com/apache/ozone/pull/4572#discussion_r1169091624
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java:
##########
@@ -373,6 +373,7 @@ private OzoneConsts() {
// For Multipart upload
public static final int OM_MULTIPART_MIN_SIZE = 5 * 1024 * 1024;
+ public static final int OM_MULTIPART_MAX_PART_NUMBER = 10000;
Review Comment:
## Test Result
**The memory usage after 10000 "part key" upload.**
```shell
[root@centos /root/]% jmap -histo:live `jps | grep OzoneManagerStarter | awk
'{ print $1 }'` | head -n 20
num #instances #bytes class name
----------------------------------------------
1: 46497817 1859912680 java.util.TreeMap$Entry
1: 13112075 419586400 java.util.HashMap$Node
2: 558238 78747928 [B
3: 444650 34326104 [C
4: 268746 14390832 [Ljava.lang.Object;
5: 260101 12484848
org.apache.hadoop.hdds.protocol.proto.HddsProtos$Port
6: 486330 11671920 com.google.protobuf.LiteralByteString
7: 443954 10654896 java.lang.String
8: 330385 10572320 org.apache.hadoop.ozone.OzoneAcl
9: 330434 7932848 [J
10: 330388 7929312 java.util.BitSet
11: 294524 7068576 java.util.ArrayList
12: 104041 5826296
org.apache.hadoop.hdds.protocol.proto.HddsProtos$UUID
13: 22023 5814072
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMResponse
14: 72411 5639624 [Ljava.util.HashMap$Node;
15: 220320 5287680
org.apache.hadoop.hdds.protocol.DatanodeDetails$Port
16: 101153 4855344 java.util.HashMap
17: 148143 4740576 java.util.UUID
```
**After the upload of 10000 parts, 2G of `ParOldGen` memory will be
generated**
```shell
[root@centos /root/]% jcmd `jps | grep OzoneManagerStarter | awk '{ print
$1 }'` GC.heap_info
33896:
PSYoungGen total 691712K, used 448768K [0x000000076ab00000,
0x000000079df80000, 0x00000007c0000000)
eden space 566784K, 78% used
[0x000000076ab00000,0x0000000785faef78,0x000000078d480000)
from space 124928K, 1% used
[0x0000000796580000,0x0000000796711278,0x000000079df80000)
to space 136704K, 0% used
[0x000000078d480000,0x000000078d480000,0x0000000795a00000)
ParOldGen total 2796544K, used 2115856K [0x00000006c0000000,
0x000000076ab00000, 0x000000076ab00000)
object space 2796544K, 75% used
[0x00000006c0000000,0x0000000741244250,0x000000076ab00000)
Metaspace used 57872K, capacity 60290K, committed 60800K, reserved 1103872K
class space used 6552K, capacity 6955K, committed 7040K, reserved 1048576K
```
---
## Reson
get OmMultipartKeyInfo copies out a new copy each time, so 10000 is only one
memory consumption
get OmMultipartKeyInfo
https://github.com/apache/ozone/blob/cffa3864b974499078fbdb37506ed992319b09c5/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java#L149-L150
copies out a new copy
https://github.com/apache/ozone/blob/cffa3864b974499078fbdb37506ed992319b09c5/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TypedTable.java#L176
**So, If we have 10,000 parts when multiupload, there will create (1 +
10000) * 10000 / 2 = 50005000 `TreeMap.Entry`**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
