[
https://issues.apache.org/jira/browse/HDDS-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17285988#comment-17285988
]
Rakesh Radhakrishnan edited comment on HDDS-4813 at 3/9/21, 9:19 AM:
---------------------------------------------------------------------
Adding few details about the DB table level operations on S3 multipart request
with an example for future reference.
KeyName = "a/b/c/file1".
Assume there are three parts, which will be having part number assigned like,
1, 2 and 3.
*User Operation-1)* S3InitiateMultipartUpload : KeyName="a/b/c/file1"
{code:java}
1) UploadID will be generated by OM : UUID.randomUUID().toString() +
"-" + UniqueId.next()
2) Add "a/b/c" parent dirs into DirectoryTable
3) Add entry to openFileTable:
TableKey = multipartKey, which is nothing but
<parentId/fileName/uploadId>
TableValue = omKeyInfo, which will store fileName and parentID.
4) Add entry to multipartFileInfoTable:
TableKey = multipartKey, which is nothing but
<parentId/fileName/uploadId>
TableValue = omKeyInfo, which will store uploadID and parentID.
Since there is no real upload started, partKeyInfoList will be empty.
Say, uploadID = 12345-0000. This is the unique identifier for uploading
parts for the corresponding keyname.
{code}
*User Operation-2)* S3CreateMultipartKey : KeyName="a/b/c/file1", PartNumber=1,
uploadID="12345-0000"
{code:java}
1) ClientID will be generated by OM : UniqueId.next()
2) Since it has uploadID, it will check OmKeyInfo in openFileTable.
2) Add entry to OpenFileTable:
TableKey = openFileKey, which is nothing but
<parentId/fileName/clientID>
TableValue = omKeyInfo, which will store fileName and parentID.
This returns KeyOutputStream with unique ClientID. Say, ClientID=9777
and user streams data to it.
Finally during close, will invoke commitMultipartUploadPart.
{code}
*User Operation-3)* S3CommitMultipartUploadPart: KeyName="a/b/c/file1",
PartNumber=1, uploadID="12345-0000", ClientID=9777.
{code:java}
1) Get <parentId/fileName/clientID> entry from openFileTable. Say, we
got OmKeyInfo1 object from DB.
2) Create 'PartKeyInfo' using the OmKeyInfo1 object.
PartNumber = 1
PartName = 1027/file1 + 9777; // <parentID>/fileName +
ClientID;
keyInfo = OmKeyInfo1 // here it stores only the fileName
and parentID.
3) Get <parentId/fileName/uploadId> entry from multipartFileInfoTable.
Say, we got multipartKeyInfo1 object from DB.
4) Add 'PartKeyInfo' to multipartKeyInfo1 object like,
multipartKeyInfo.addPartKeyInfo(partNumber,
partKeyInfo.build());
5) Add the above multipartKeyInfo into the multipartFileInfoTable.
6) Delete <parentId/fileName/clientID> entry from openFileTable.
{code}
*User Operation-4)*
{code:java}
Repeat S3CreateMultipartKey and S3CommitMultipartUploadPart for
PartNumber=2 and PartNumber=3.
After this operation, now the 'multipartKeyInfo' entry in
multipartFileInfoTable will have three 'PartKeyInfo' items in the list.
{code}
*User Operation-5)* S3CompleteMultipartUpload: KeyName="a/b/c/file1",
partsMap=[1,2,3], uploadID="12345-0000".
{code:java}
1) Get <parentId/fileName/uploadId> entry from multipartFileInfoTable.
Say, we got multipartKeyInfo1 object from DB.
2) Compare all the user given part numbers and data size calculation
etc using all the parts. I'm not exploring those info now as it is existing
logic.
3) Prepare OmKeyInfo using <parentId/fileName> and store entry into
FileTable. This is now the reference to the KeyName="a/b/c/file1".
4) Delete <parentId/fileName/uploadId> entry from openFileTable.
5) Delete <parentId/fileName/uploadId> entry from
multipartFileInfoTable.
{code}
was (Author: rakeshr):
Adding few details about the DB table level operations on S3 multipart request
with an example for future reference.
KeyName = "a/b/c/file1".
Assume there are three parts, which will be having part number assigned like,
1, 2 and 3.
*User Operation-1)* S3InitiateMultipartUpload : KeyName="a/b/c/file1"
{code}
1) UploadID will be generated by OM : UUID.randomUUID().toString() +
"-" + UniqueId.next()
2) Add "a/b/c" parent dirs into DirectoryTable
3) Add entry to openFileTable:
TableKey = multipartKey, which is nothing but
<parentId/fileName/uploadId>
TableValue = omKeyInfo, which will store fileName and parentID.
4) Add entry to multipartFileInfoTable:
TableKey = multipartKey, which is nothing but
<parentId/fileName/uploadId>
TableValue = omKeyInfo, which will store uploadID and parentID.
Since there is no real upload started, partKeyInfoList will be empty.
Say, uploadID = 12345-0000. This is the unique identifier for uploading
parts for the corresponding keyname.
{code}
*User Operation-2)* S3CreateMultipartKey : KeyName="a/b/c/file1", PartNumber=1,
uploadID="12345-0000"
{code}
1) ClientID will be generated by OM : UniqueId.next()
2) Since it has uploadID, it will check OmKeyInfo in openFileTable.
2) Add entry to OpenFileTable:
TableKey = openFileKey, which is nothing but
<parentId/fileName/clientID>
TableValue = omKeyInfo, which will store fileName and parentID.
This returns KeyOutputStream with unique ClientID. Say, ClientID=9777
and user streams data to it.
Finally during close, will invoke commitMultipartUploadPart.
{code}
*User Operation-3)* S3CommitMultipartUploadPart: KeyName="a/b/c/file1",
PartNumber=1, uploadID="12345-0000", ClientID=9777.
{code}
1) Get <parentId/fileName/clientID> entry from openFileTable. Say, we
got OmKeyInfo1 object from DB.
2) Create 'PartKeyInfo' using the OmKeyInfo1 object.
PartNumber = 1
PartName = file1 + 9777; // fileName + ClientID;
keyInfo = OmKeyInfo1 // here it stores only the fileName
and parentID.
3) Get <parentId/fileName/uploadId> entry from multipartFileInfoTable.
Say, we got multipartKeyInfo1 object from DB.
4) Add 'PartKeyInfo' to multipartKeyInfo1 object like,
multipartKeyInfo.addPartKeyInfo(partNumber,
partKeyInfo.build());
5) Add the above multipartKeyInfo into the multipartFileInfoTable.
6) Delete <parentId/fileName/clientID> entry from openFileTable.
{code}
*User Operation-4)*
{code}
Repeat S3CreateMultipartKey and S3CommitMultipartUploadPart for
PartNumber=2 and PartNumber=3.
After this operation, now the 'multipartKeyInfo' entry in
multipartFileInfoTable will have three 'PartKeyInfo' items in the list.
{code}
*User Operation-5)* S3CompleteMultipartUpload: KeyName="a/b/c/file1",
partsMap=[1,2,3], uploadID="12345-0000".
{code}
1) Get <parentId/fileName/uploadId> entry from multipartFileInfoTable.
Say, we got multipartKeyInfo1 object from DB.
2) Compare all the user given part numbers and data size calculation
etc using all the parts. I'm not exploring those info now as it is existing
logic.
3) Prepare OmKeyInfo using <parentId/fileName> and store entry into
FileTable. This is now the reference to the KeyName="a/b/c/file1".
4) Delete <parentId/fileName/uploadId> entry from openFileTable.
5) Delete <parentId/fileName/uploadId> entry from
multipartFileInfoTable.
{code}
> [FSO]S3Multipart: Implement UploadCompleteRequest
> -------------------------------------------------
>
> Key: HDDS-4813
> URL: https://issues.apache.org/jira/browse/HDDS-4813
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Rakesh Radhakrishnan
> Assignee: Rakesh Radhakrishnan
> Priority: Major
> Labels: pull-request-available
>
> This task is to implement prefix based FSO for the
> {{S3MultipartUploadCompleteRequest}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
