[
https://issues.apache.org/jira/browse/HADOOP-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17929533#comment-17929533
]
ASF GitHub Bot commented on HADOOP-15224:
-----------------------------------------
raphaelazzolini commented on PR #7396:
URL: https://github.com/apache/hadoop/pull/7396#issuecomment-2677228631
## Performance Tests
I executed the performance tests with all four checksum algorithms and no
checksum. The results are in
[cloudstore-test.zip](https://github.com/user-attachments/files/18934566/cloudstore-test.zip).
These tests are using blocks of 64 MB.
These are tests using blocks of 256 MB:
[cloudstore-test-256mb.zip](https://github.com/user-attachments/files/18934689/cloudstore-test-256mb.zip)
## Delete Throttling
You can see the logs in
[failsafe-reports.zip](https://github.com/user-attachments/files/18934589/failsafe-reports.zip).
I tried to create the surefire report html, but it was empty, not sure what I
did wrong.
It looks like we get a 503, but somewhere is making the
AwsXmlPredicatedResponseHandler to translate it to 200.
About the error message, it may not be frozen, but I think it is very hard
that it would be changed because it is not something backwards compatible to
do. However, relying on the error code would be a much better option.
> builld up md5 checksum as blocks are built in S3ABlockOutputStream; validate
> upload
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-15224
> URL: https://issues.apache.org/jira/browse/HADOOP-15224
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Raphael Azzolini
> Priority: Minor
> Labels: pull-request-available
>
> [~rdblue] reports sometimes he sees corrupt data on S3. Given MD5 checks from
> upload to S3, its likelier to have happened in VM RAM, HDD or nearby.
> If the MD5 checksum for each block was built up as data was written to it,
> and checked against the etag RAM/HDD storage of the saved blocks could be
> removed as sources of corruption
> The obvious place would be
> {{org.apache.hadoop.fs.s3a.S3ADataBlocks.DataBlock}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
