[
https://issues.apache.org/jira/browse/HADOOP-19654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18032862#comment-18032862
]
Steve Loughran commented on HADOOP-19654:
-----------------------------------------
AWS SDK issue #6518 shows how checksum generation on uploaded data
(fs.s3a.create.checksum) must be set if request checksum calculation is enabled
(fs.s3a.checksum.generation)
Checksum validation has also been enabled by default;
{{ITestS3AOpenCost.testStreamIsNotChecksummed()}} caught that change.
It looks like the SDK has really embraced checksums, which first broke
compatibility with other stores, but which has also surfaced problems within
their own code.
All checksum logic will be off by default; MD5 headers will be attached now
> Upgrade AWS SDK to 2.35.4
> -------------------------
>
> Key: HADOOP-19654
> URL: https://issues.apache.org/jira/browse/HADOOP-19654
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, fs/s3
> Affects Versions: 3.5.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Upgrade to a recent version of 2.33.x or later while off the critical path of
> things.
> HADOOP-19485 froze the sdk at a version which worked with third party stores.
> Apparently the new version works; early tests show that Bulk Delete calls
> with third party stores complain about lack of md5 headers, so some tuning is
> clearly going to be needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
