[
https://issues.apache.org/jira/browse/HDDS-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Sizov updated HDDS-11223:
--------------------------------
Description:
*Steps to reproduce*
- Enable hdds.datanode.chunk.data.validation.check property.
- put any file into a EC bucket
The output a client gets is:
{noformat}
INTERNAL_ERROR No enough datanodes to choose. TotalNodes = 10 AvailableNodes =
4 RequiredNodes = 5 ExcludedNodes = 6 UsedNodes = 0
{noformat}
*Investigation details*
The issue comes from incorrect checksum validation on the datanode side.
Here are the details:
1. {{KeyValueHandler.validateChunkChecksumData}} calls
{{Checksum.verifyChecksum}}
2. {{Checksum.verifyChecksum}} calls {{checksum.computeChecksum(data)}} to
calculate a checksum over the received data
3. {{checksum.computeChecksum}} has the following block:
{code}
for (ByteBuffer b : data.iterate(bytesPerChecksum)) {
checksumList.add(computeChecksum(b, function, bytesPerChecksum));
}
{code}
The issue is that {{data.iterate()}} may return byte buffers of an arbitrary
size.
Specifically, it is observed on {{ChunkBufferImplWithByteBufferList}} which was
created from a {{RopeByteString}} in {{KeyValueHandler.handleWriteChunk}}:
{code}
data =
ChunkBuffer.wrap(writeChunk.getData().asReadOnlyByteBufferList());
{code}
So checksums rely on the fact the the requested data buffer is split into parts
of the requested size (and a smaller final one),
but {{ChunkBufferImplWithByteBufferList}} for some reason declares the opposite.
{code}
/**
* Returns the next buffer in the list irrespective of the bufferSize.
*/
@Override
public Iterable<ByteBuffer> iterate(int bufferSize) {
{code}
was:
*Steps to reproduce*
- Enable hdds.datanode.chunk.data.validation.check property.
- put any file into a EC bucket
The output a client gets is:
{noformat}
INTERNAL_ERROR No enough datanodes to choose. TotalNodes = 10 AvailableNodes =
4 RequiredNodes = 5 ExcludedNodes = 6 UsedNodes = 0
{noformat}
*Investigation details*
The issue comes from incorrect checksum validation on the datanode side.
Here are the details:
1. {{KeyValueHandler.validateChunkChecksumData}} calls
{{Checksum.verifyChecksum}}
2. {{Checksum.verifyChecksum}} calls {{checksum.computeChecksum(data)}} to
calculate a checksum over the received data
3. {{checksum.computeChecksum}} has the following block:
{code}
for (ByteBuffer b : data.iterate(bytesPerChecksum)) {
checksumList.add(computeChecksum(b, function, bytesPerChecksum));
}
{code}
The issue is that {{data.iterate()}} may return byte buffers of an arbitrary
size.
Specifically, it is observed on {{ChunkBufferImplWithByteBufferList}} which was
created from a {{RopeByteString}} in {{KeyValueHandler.handleWriteChunk}}:
{code}
data =
ChunkBuffer.wrap(writeChunk.getData().asReadOnlyByteBufferList());
{code}
> Datanode checksum validation in EC bucket does not work
> -------------------------------------------------------
>
> Key: HDDS-11223
> URL: https://issues.apache.org/jira/browse/HDDS-11223
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Kirill Sizov
> Priority: Major
>
> *Steps to reproduce*
> - Enable hdds.datanode.chunk.data.validation.check property.
> - put any file into a EC bucket
> The output a client gets is:
> {noformat}
> INTERNAL_ERROR No enough datanodes to choose. TotalNodes = 10 AvailableNodes
> = 4 RequiredNodes = 5 ExcludedNodes = 6 UsedNodes = 0
> {noformat}
> *Investigation details*
> The issue comes from incorrect checksum validation on the datanode side.
> Here are the details:
> 1. {{KeyValueHandler.validateChunkChecksumData}} calls
> {{Checksum.verifyChecksum}}
> 2. {{Checksum.verifyChecksum}} calls {{checksum.computeChecksum(data)}} to
> calculate a checksum over the received data
> 3. {{checksum.computeChecksum}} has the following block:
> {code}
> for (ByteBuffer b : data.iterate(bytesPerChecksum)) {
> checksumList.add(computeChecksum(b, function, bytesPerChecksum));
> }
> {code}
> The issue is that {{data.iterate()}} may return byte buffers of an arbitrary
> size.
> Specifically, it is observed on {{ChunkBufferImplWithByteBufferList}} which
> was created from a {{RopeByteString}} in {{KeyValueHandler.handleWriteChunk}}:
> {code}
> data =
> ChunkBuffer.wrap(writeChunk.getData().asReadOnlyByteBufferList());
> {code}
> So checksums rely on the fact the the requested data buffer is split into
> parts of the requested size (and a smaller final one),
> but {{ChunkBufferImplWithByteBufferList}} for some reason declares the
> opposite.
> {code}
> /**
> * Returns the next buffer in the list irrespective of the bufferSize.
> */
> @Override
> public Iterable<ByteBuffer> iterate(int bufferSize) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
