xichen01 opened a new pull request, #5524:
URL: https://github.com/apache/ozone/pull/5524
## What changes were proposed in this pull request?
To fix the problem of abnormal dirty data overwriting normal data when S3G
writes data. Make Ozone S3 puts atomic writes, all successes total or failures
The main implementation is to check whether the data of the key is complete
or not when committing, if the data written by the key is different from the
expected size, the key is considered incomplete, and the commit operation is
not executed.
```java
@Override
public synchronized void close() throws IOException {
//...
try {
//...
if (requiresAtomicWrite) {
long expectedSize = blockOutputStreamEntryPool.getDataSize();
Preconditions.checkArgument(expectedSize == offset,
String.format("Expected: %d and actual %d write sizes do not
match",
expectedSize, offset));
}
blockOutputStreamEntryPool.commitKey(offset);
} finally {
blockOutputStreamEntryPool.cleanup();
}
}
```
- The `requiresAtomicWrite` is only `enable` in S3G, it will be set in
`EndpointBase#initialization` , just like the `s3Auth`
- The `blockOutputStreamEntryPool.getDataSize()` is the `dataSize` that was
passed in when the S3G created the key (including the MPU key).
- The dataSize is the
- `Content-Length`: normal put key
- `x-amz-decoded-content-length`: Stream mode uploaded key.
- `x-amz-copy-source-range`: The length specified in the MPU copy Range
(Ozone does not currently support Copy Ranges for non-MPU keys.).
PS:
refer to AWS put doc
https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
> Amazon S3 never adds partial objects; if you receive a success response,
Amazon S3 added the entire object to the bucket.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9526
## How was this patch tested?
unit test, manually test.
manually test.
```bash
[root@VM-8-3-centos ~]$ aws configure set default.s3.multipart_threshold 1GB
[root@VM-8-3-centos ~]$ aws s3 --endpoint-url http://localhost:9878 cp
~/500M.img s3://bucket1/500M.img
^Ccancelled: ctrl-c received
[root@VM-8-3-centos /tmp]$ aws s3 ls s3://bucket1/ --endpoint-url
http://localhost:9878
// A key that has not been completely written should not be visible.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
