[
https://issues.apache.org/jira/browse/HDDS-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138974#comment-17138974
]
Sammi Chen edited comment on HDDS-3512 at 6/18/20, 3:16 AM:
------------------------------------------------------------
It turns out that aws sdk 1.11.* by default send the multi-upload content as
the following format, so the extra size is the chunk-signature.
{code:java}
n;chunk-signature=<hash>
real data
n;chunk-signature=<hash>
{code}
In this case, the "Content-Length" header represents data+signature size, the
“x-amz-decoded-content-length” header represents the real data size.
Reference https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
The desired solution is for Ozone s3g to supports this new format. It requires
time.
Temporarily, I figured out a solution at client side to disable this new
format. Here is the reference piece of code，withChunkedEncodingDisabled(true))
is the key point.
{code:java}
s3 = new AmazonS3Client(provider, awsConf);
s3.setS3ClientOptions(new
S3ClientOptions().withPathStyleAccess(true).withChunkedEncodingDisabled(true));
s3.setEndpoint(endpoint);
xfer_mgr = new TransferManager(s3);
{code}
was (Author: sammi):
It turns out that aws sdk 1.11.* by default send the multi-upload content as
the following format, so the extra size is the chunk-signature.
{code:java}
n;chunk-signature=<hash>
real data
n;chunk-signature=<hash>
{code}
In this case, the "Content-Length" header represents data+signature size, the
“x-amz-decoded-content-length” header represents the real data size.
Reference https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
The desired solution is for Ozone s3g to supports this new format. It requires
time.
Temporarily, I figured out a solution at client side to disable this new
format. Here is the reference piece of code，withChunkedEncodingDisabled(true))
is the key point.
{code:java}
s3 = new AmazonS3Client(provider, awsConf);
s3.setS3ClientOptions(new
S3ClientOptions().withPathStyleAccess(true).withChunkedEncodingDisabled(true));
s3.setEndpoint(endpoint);
xfer_mgr = new TransferManager(s3);
{code}
> s3g multi-upload saved content incorrect when client uses aws java sdk 1.11.*
> jar
> ---------------------------------------------------------------------------------
>
> Key: HDDS-3512
> URL: https://issues.apache.org/jira/browse/HDDS-3512
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
> Reporter: Sammi Chen
> Assignee: Marton Elek
> Priority: Blocker
> Labels: TriagePending
>
> The default multi-part size is 5MB, which is 5242880 byte, while all the
> chunks saved by s3g is 5246566 byte which is greater than 5MB.
> By looking into the ObjectEndpoint.java, it seems the chunk size is retrieved
> from the "Content-Length" header.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
