[
https://issues.apache.org/jira/browse/HDDS-10395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-10395:
----------------------------------
Labels: pull-request-available (was: )
> Fix compatibility issue with eTag during MPU listParts
> ------------------------------------------------------
>
> Key: HDDS-10395
> URL: https://issues.apache.org/jira/browse/HDDS-10395
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM, Ozone Manager, S3
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
> Labels: pull-request-available
>
> Found an eTag incompatibility in S3 listParts while testing the multipart
> uploads old S3G and new OMs.
> First issue is in KeyManagerImpl#listParts
> {code:java}
> OmPartInfo omPartInfo = new OmPartInfo(partKeyInfo.getPartNumber(),
> partName,
> partKeyInfo.getPartKeyInfo().getModificationTime(),
> partKeyInfo.getPartKeyInfo().getDataSize(),
> partKeyInfo.getPartKeyInfo().getMetadataList().stream()
> .filter(keyValue -> keyValue.getKey().equals(ETAG))
> .findFirst().get().getValue()); {code}
> This will throw "java.util.NoSuchElementException: No value present" in case
> where the MPU part does not contain eTag field (before HDDS-9680)
> Second issue, is that ObjectEndpoint#listParts is currently only returning
> the MPU part eTag, which might not exist for old MPU parts. This can be
> resolved by falling back to using partName as eTag if the eTag is not
> specified.
> Third issue is the NPE when calling setETag with null (in
> OmPartInfo#getProto). This can be resolved by doing a simple nullity check.
> For reference, the issue replicated with the following MPU script which
> manually call the s3 API for MPU. It should also be able to be replicated
> with "aws s3 cp" for large files since it will use multipart uploads, and
> most probably will call the listParts API.
>
> {code:java}
> #!/bin/bash
> BUCKET_NAME="etag-test-bucket"
> KEY_NAME="mpu-key"
> ENDPOINT_URL="S3_ENDPOINT"
> AWS_ACCESS_KEY_ID="ACCESS_KEY_ID"
> AWS_SECRET_ACCESS_KEY="SECRET_ACCESS_KEY"
> # Create three files
> dd if=/dev/urandom of=/tmp/part1 bs=1M count=10
> dd if=/dev/urandom of=/tmp/part2 bs=1M count=10
> dd if=/dev/urandom of=/tmp/part3 bs=1M count=10# Define a function to
> conditionally add the endpoint url
> function aws_cmd {
> if [[ -z "$ENDPOINT_URL" ]]; then
> AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
> AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY aws s3api $@
> else
> AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
> AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY aws s3api --endpoint-url
> $ENDPOINT_URL $@
> fi
> }
> # Start the multipart upload and get the upload ID
> UPLOAD_ID=$(aws_cmd create-multipart-upload --bucket $BUCKET_NAME --key
> $KEY_NAME --query 'UploadId' --output text)
> echo "upload id: $UPLOAD_ID"# Upload the parts
> ETAG1=$(aws_cmd upload-part --bucket $BUCKET_NAME --key $KEY_NAME
> --part-number 1 --body /tmp/part1 --upload-id $UPLOAD_ID --query 'ETag'
> --output text)
> echo "ETAG1: $ETAG1"
> ETAG2=$(aws_cmd upload-part --bucket $BUCKET_NAME --key $KEY_NAME
> --part-number 2 --body /tmp/part2 --upload-id $UPLOAD_ID --query 'ETag'
> --output text)
> echo "ETAG2: $ETAG2"
> ETAG3=$(aws_cmd upload-part --bucket $BUCKET_NAME --key $KEY_NAME
> --part-number 3 --body /tmp/part3 --upload-id $UPLOAD_ID --query 'ETag'
> --output text)
> echo "ETAG3: $ETAG3"
> # List the MPU parts (the issue was detected here)
> aws_cmd list-parts --bucket $BUCKET_NAME --key $KEY_NAME --upload-id
> $UPLOAD_ID
> aws_cmd complete-multipart-upload --multipart-upload
> "Parts=[{ETag=$ETAG1,PartNumber=1},{ETag=$ETAG2,PartNumber=2},{ETag=$ETAG3,PartNumber=3}]"
> --bucket $BUCKET_NAME --key $KEY_NAME --upload-id $UPLOAD_ID
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
