ivandika3 opened a new pull request, #6235:
URL: https://github.com/apache/ozone/pull/6235
## What changes were proposed in this pull request?
Found an eTag incompatibility in S3 listParts while testing the multipart
uploads old S3G and new OMs.
First issue is in KeyManagerImpl#listParts
```java
OmPartInfo omPartInfo = new OmPartInfo(partKeyInfo.getPartNumber(),
partName,
partKeyInfo.getPartKeyInfo().getModificationTime(),
partKeyInfo.getPartKeyInfo().getDataSize(),
partKeyInfo.getPartKeyInfo().getMetadataList().stream()
.filter(keyValue -> keyValue.getKey().equals(ETAG))
.findFirst().get().getValue());
```
This will throw "java.util.NoSuchElementException: No value present" in case
where the MPU part does not contain eTag field (before
[HDDS-9680](https://issues.apache.org/jira/browse/HDDS-9680))
Second issue, is that ObjectEndpoint#listParts is currently only returning
the MPU part eTag, which might not exist for old MPU parts. This can be
resolved by falling back to using partName as eTag if the eTag is not specified.
Third issue is the NPE when calling setETag with null (in
OmPartInfo#getProto). This can be resolved by doing a simple nullity check.
For reference, the issue replicated with the following MPU script which
manually call the s3 API for MPU. It should also be able to be replicated with
"aws s3 cp" for large files since it will use multipart uploads, and most
probably will call the listParts API.
```bash
#!/bin/bash
BUCKET_NAME="etag-test-bucket"
KEY_NAME="mpu-key"
ENDPOINT_URL="S3_ENDPOINT"
AWS_ACCESS_KEY_ID="ACCESS_KEY_ID"
AWS_SECRET_ACCESS_KEY="SECRET_ACCESS_KEY"
# Create three files
dd if=/dev/urandom of=/tmp/part1 bs=1M count=10
dd if=/dev/urandom of=/tmp/part2 bs=1M count=10
dd if=/dev/urandom of=/tmp/part3 bs=1M count=10# Define a function to
conditionally add the endpoint url
function aws_cmd {
if [[ -z "$ENDPOINT_URL" ]]; then
AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY aws s3api $@
else
AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY aws s3api --endpoint-url
$ENDPOINT_URL $@
fi
}
# Start the multipart upload and get the upload ID
UPLOAD_ID=$(aws_cmd create-multipart-upload --bucket $BUCKET_NAME --key
$KEY_NAME --query 'UploadId' --output text)
echo "upload id: $UPLOAD_ID"# Upload the parts
ETAG1=$(aws_cmd upload-part --bucket $BUCKET_NAME --key $KEY_NAME
--part-number 1 --body /tmp/part1 --upload-id $UPLOAD_ID --query 'ETag'
--output text)
echo "ETAG1: $ETAG1"
ETAG2=$(aws_cmd upload-part --bucket $BUCKET_NAME --key $KEY_NAME
--part-number 2 --body /tmp/part2 --upload-id $UPLOAD_ID --query 'ETag'
--output text)
echo "ETAG2: $ETAG2"
ETAG3=$(aws_cmd upload-part --bucket $BUCKET_NAME --key $KEY_NAME
--part-number 3 --body /tmp/part3 --upload-id $UPLOAD_ID --query 'ETag'
--output text)
echo "ETAG3: $ETAG3"
# List the MPU parts (the issue was detected here)
aws_cmd list-parts --bucket $BUCKET_NAME --key $KEY_NAME --upload-id
$UPLOAD_ID
aws_cmd complete-multipart-upload --multipart-upload
"Parts=[{ETag=$ETAG1,PartNumber=1},{ETag=$ETAG2,PartNumber=2},{ETag=$ETAG3,PartNumber=3}]"
--bucket $BUCKET_NAME --key $KEY_NAME --upload-id $UPLOAD_ID
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10395
## How was this patch tested?
Manual testing between OM and S3G of different versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
