[
https://issues.apache.org/jira/browse/HDDS-8238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18059517#comment-18059517
]
Tsz-wo Sze commented on HDDS-8238:
----------------------------------
[~abhishek.pal], thanks a lot for picking this up!
> Multipart upload generates a huge OM rocksdb log
> ------------------------------------------------
>
> Key: HDDS-8238
> URL: https://issues.apache.org/jira/browse/HDDS-8238
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM
> Reporter: Tsz-wo Sze
> Assignee: Abhishek Pal
> Priority: Major
> Labels: ozone-performance
>
> When using multipart upload to upload a 1000-part key. OM RocksDb log needs
> to use 733.15 MB disk space. For a 10,000-part key, it can use 71.5 GB . The
> disk space usage for the log is O(n^2) for a n-part key.
> MultipartKeyInfo has a list of PartKeyInfo (partKeyInfoList) while each
> PartKeyInfo has a KeyInfo and the size of each KeyInfo is ~1.5 KB.
> {code:java}
> //OmClientProtocol.proto
> message MultipartKeyInfo {
> required string uploadID = 1;
> ...
> repeated PartKeyInfo partKeyInfoList = 5;
> ...
> }
> message PartKeyInfo {
> required string partName = 1;
> required uint32 partNumber = 2;
> required KeyInfo partKeyInfo = 3; // the size of a KeyInfo is ~1.5 KB
> }
> {code}
> Multipart upload repeatedly puts the same key with a MultipartKeyInfo object
> for each part.
> * When uploading the first part, partKeyInfoList has 1 element.
> * When uploading the second part, partKeyInfoList has 2 elements.
> * When uploading the third part, partKeyInfoList has 3 elements.
> ...
> * When uploading the 1000th part, partKeyInfoList has 1000 elements.
> The partKeyInfoList is written to RocksDb log for each part. The number of
> PartKeyInfo objects written to RocksDb log is
> - (1 + 1000)*1000/2 = 500,500 objects
> and the size is 1.5 KB * 500,500 = 733.15 MB.
> Note that only the RocksDb log size is large. The size of the RocksDb tables
> are small since the partKeyInfoList will be overwritten again and again for
> each part.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
