ivandika3 opened a new pull request, #6377:
URL: https://github.com/apache/ozone/pull/6377
## What changes were proposed in this pull request?
A user encountered this error when it tries to download a key using the AWS
S3 SDK (version 1.11.415).
```
ERROR FileOperationServiceImpl - s3Client.getObject invoke error,
objectKey:<redacted>. (FileOperationServiceImpl.java:438)
java.lang.ArrayIndexOutOfBoundsException: 110
at com.amazonaws.util.Base16Codec.pos(Base16Codec.java:96)
at com.amazonaws.util.Base16Codec.decode(Base16Codec.java:87)
at com.amazonaws.util.Base16Lower.decode(Base16Lower.java:53)
at com.amazonaws.util.BinaryUtils.fromHex(BinaryUtils.java:48)
at
com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1456)
...
```
Although the key was able to be downloaded using AWS s3api
```
aws s3api --endpoint <redacted> get-object --bucket <redacted> --key
<redacted> download.txt
{
"AcceptRanges": "bytes",
"LastModified": "Wed, 28 Feb 2024 11:06:30 GMT",
"ContentLength": 1117055,
"ETag": "\"null\"",
"CacheControl": "no-cache",
"ContentType": "application/octet-stream",
"Expires": "Wed, 13 Mar 2024 08:10:05 GMT",
"Metadata": {}
}
```
The problem can be replicated even with the current version by uploading a
file with ofs and downloading it using AWS S3 SDK.
It seems object without ETag field was able to be downloaded using AWS CLI,
but not AWS SDK.
After looking at the AWS SDK code it seems that AWS SDK will do a
post-processing step that will validate the ETag field of the downloaded object
to the object's content. If the ETag field is null, the post-processing step
will skip the validation.
Currently, S3G returns a string "null" for the ETag field if the ETag field
does not exist, which should cause the AWS SDK to not be able to parse the
string since it md5 string is longer than the "null" string. This is most
probably why there is an ArrayIndexOutOfBoundsException
One possible solution is to not return the ETag field at all if the key does
not contain ETag to begin with. This way the post processing step in the AWS
SDK will not validate the md5 hash.
Future improvement: We might need to add some test coverage for AWS SDK
(maybe in the integration tests) since the behavior might be different than AWS
CLI.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10521
## How was this patch tested?
Manual tests: Creating a S3 key with ofs and downloading it with the AWS
Java SDK GetObject.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
