rakeshadr commented on a change in pull request #1923:
URL: https://github.com/apache/ozone/pull/1923#discussion_r577677611
##########
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequestV1.java
##########
@@ -147,8 +148,18 @@ public OMClientResponse
validateAndUpdateCache(OzoneManager ozoneManager,
// Set the UpdateID to current transactionLogIndex
omKeyInfo.setUpdateID(trxnLogIndex, ozoneManager.isRatisEnabled());
- String ozoneKey = omMetadataManager.getOzonePathKey(parentID, fileName);
- partName = ozoneKey + clientID;
+ /**
+ * Format of PartName stored into MultipartInfoTable is,
+ * "fileName + ClientID".
+ *
+ * Contract is that all part names present in a multipart info will
+ * have same key prefix path.
+ *
+ * For example:
+ * /vol1/buck1/a/b/c/part-1, /vol1/buck1/a/b/c/part-2,
+ * /vol1/buck1/a/b/c/part-n
+ */
+ dbPartName = fileName + clientID;
Review comment:
@linyiqun Yes, there will be multiple parts during the upload
operations. Finally openFileTable, multiPartFileInfoTable entries will be
deleted and then add fileTable entry corresponding to the user given KeyName. I
have tried adding internals especially focussing on DB operations with an
example for better understanding, can you please go through it and hope that
helps to proceed further. Thanks a lot!
KeyName = "a/b/c/file1".
Assume there are three parts, which will be having part number assigned
like, 1, 2 and 3.
**User Operation-1)** S3InitiateMultipartUpload : KeyName="a/b/c/file1"
```
1) UploadID will be generated by OM : UUID.randomUUID().toString() +
"-" + UniqueId.next()
2) Add "a/b/c" parent dirs into DirectoryTable
3) Add entry to openFileTable:
TableKey = multipartKey, which is nothing but
<parentId/fileName/uploadId>
TableValue = omKeyInfo, which will store fileName and
parentID.
4) Add entry to multipartFileInfoTable:
TableKey = multipartKey, which is nothing but
<parentId/fileName/uploadId>
TableValue = omKeyInfo, which will store uploadID and
parentID. Since there is no real upload started, partKeyInfoList will be empty.
Say, uploadID = 12345-0000. This is the unique identifier for uploading
parts for the corresponding keyname.
```
**User Operation-2)** S3CreateMultipartKey : KeyName="a/b/c/file1",
PartNumber=1, uploadID="12345-0000"
```
1) ClientID will be generated by OM : UniqueId.next()
2) Since it has uploadID, it will check OmKeyInfo in openFileTable.
2) Add entry to OpenFileTable:
TableKey = openFileKey, which is nothing but
<parentId/fileName/clientID>
TableValue = omKeyInfo, which will store fileName and
parentID.
This returns KeyOutputStream with unique ClientID. Say, ClientID=9777
and user streams data to it.
Finally during close, will invoke commitMultipartUploadPart.
```
**User Operation-3)** S3CommitMultipartUploadPart: KeyName="a/b/c/file1",
PartNumber=1, uploadID="12345-0000", ClientID=9777
```
1) Get <parentId/fileName/clientID> entry from openFileTable. Say, we
got OmKeyInfo1 object from DB.
2) Create 'PartKeyInfo' using the OmKeyInfo1 object.
PartNumber = 1
PartName = file1 + 9777; // fileName + ClientID;
keyInfo = OmKeyInfo1 // here it stores only the
fileName and parentID.
3) Get multipartKey <parentId/fileName/uploadId> entry from
multipartFileInfoTable. Say, we got multipartKeyInfo1 object from DB.
4) Append/Add 'PartKeyInfo' to multipartKeyInfo1 object like,
multipartKeyInfo.addPartKeyInfo(partNumber,
partKeyInfo.build());
5) Update the above multipartKeyInfo into the multipartFileInfoTable.
6) Delete <parentId/fileName/clientID> entry from openFileTable.
```
**User Operation-4)**
```
Repeat S3CreateMultipartKey and S3CommitMultipartUploadPart for
PartNumber=2 and PartNumber=3.
After these three parts upload operations, now the
'multipartKeyInfo' entry in multipartFileInfoTable will have three
'PartKeyInfo' items in the list.
```
**User Operation-5)** S3CompleteMultipartUpload: KeyName="a/b/c/file1",
partsMap={1,2,3}, uploadID="12345-0000".
```
1) Get multipartKey <parentId/fileName/uploadId> entry from
multipartFileInfoTable. Say, we got multipartKeyInfo1 object from DB.
2) Compare all the user given part numbers and data size calculation
etc using all the parts. I'm not exploring those info now as it is existing
logic.
3) Prepare OmKeyInfo using <parentId/fileName> and add new entry into
FileTable. This is now become the reference to the user given
KeyName="a/b/c/file1".
4) Delete multipartKey <parentId/fileName/uploadId> entry from
openFileTable.
5) Delete multipartKey <parentId/fileName/uploadId> entry from
multipartFileInfoTable.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
