errose28 opened a new pull request, #7083:
URL: https://github.com/apache/ozone/pull/7083
## What changes were proposed in this pull request?
### Summary
A merkle tree file may be corrupted during write or by some other error.
This should only be a temporary problem since the scanner will rewrite it on
its next run. Until that happens failures should be handled accordingly:
- Clients calling the read API should be able to cleanly fail on a corrupted
file. The server does not deserialize it before sending.
- Any operation that updates the file should regenerate a local checksum
file if it is corrupted, as if it does not exist.
- File write can be done by renaming a temp file into place to avoid
corruption from partial writes. The file should either be present or absent if
a write fails, but not partially written.
### Details
- When a checksum file is corrupted, we overwrite it with a new copy. This
will lose any block delete information present in the file.
- This is not a problem because the `merge` algorithm that will be
implemented in HDDS-10928 will pick up this information again from other
replicas.
- If no other replicas have the delete information then all replicas match
and there is no problem.
- When a checksum file does not exist on the datanode, an error is returned
to the client. This prevents accidental errors that may occur from using a
default value protobuf or null value instead.
- The read checksum API now automatically fails on unclosed containers.
These containers will never have the files generated.
- Exception handling for the current read API was handled incorrectly since
the request was not marked readonly in `HddsUtils#isReadOnly`. Tests in this PR
caught the issue and it is now fixed.
- The API was renamed from `getContainerMerkleTree` to
`getContainerChecksumInfo` to match the proto value returned.
- We have protos called `ContainerMerkleTree` and `ContainerChecksumInfo`.
`ContainerChecksumInfo` contains a `ContainerMerkleTree` along with other
metadata.
- The API was called `getContaienrMerkleTree` but returned a `Bytestring`
that actually deserialized into a `ContainerChecksumInfo`, which was
unintuitive. Calling `getContainerMerkleTree` and deserializing as a
`ContainerMerkleTree` would fail.
- Since the API needs to return a `ContainerChecksumInfo` to capture all
information about the container (the merkle tree alone will not suffice), it is
intuitive to rename the API `getContainerChecksumInfo`.
## What is the link to the Apache JIRA
HDDS-11253
## How was this patch tested?
- Unit tests added for handling file corruption within the datanode.
- Integration tests added to test error propagation from server to client
when reading the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
