chungen0126 commented on code in PR #10764:
URL: https://github.com/apache/ozone/pull/10764#discussion_r3840204997
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/Checksum.java:
##########
@@ -430,4 +430,44 @@ public static void verifyChecksum(List<ByteBuffer>
bufferList, int startIndex, C
public static ContainerProtos.ChecksumData getNoChecksumDataProto() {
return new ChecksumData(ChecksumType.NONE, 0).getProtoBufMessage();
}
+
+ public static void verifyChecksum(
+ ByteBuffer data, ChecksumData checksumData, long blockOffset,
List<ContainerProtos.ChunkInfo> chunkInfoList)
+ throws OzoneChecksumException {
+ if (!checksumData.getChecksumType().equals(ChecksumType.NONE)) {
+ int bytesPerChecksum = checksumData.getBytesPerChecksum();
+ long readLength = data.remaining();
+ long currentChunkOffset = 0;
+ int checksumIndex = 0;
+ int dataOffset = 0;
+
+ for (ContainerProtos.ChunkInfo chunk : chunkInfoList) {
+ long chunkStart = currentChunkOffset;
+ long chunkEnd = chunkStart + chunk.getLen();
+
+ long overlapStart = Math.max(blockOffset, chunkStart);
+ long overlapEnd = Math.min(blockOffset + readLength, chunkEnd);
+
+ if (overlapStart < overlapEnd) {
+ int overlapLen = Math.toIntExact(overlapEnd - overlapStart);
+ ByteBuffer chunkData = data.duplicate();
+ chunkData.position(data.position() + dataOffset);
+ chunkData.limit(data.position() + dataOffset + overlapLen);
+
+ Checksum.verifyChecksum(chunkData, checksumData, checksumIndex);
+
+ dataOffset += overlapLen;
+
+ long offsetInChunk = overlapStart - chunkStart;
+ long endOffsetInChunk = overlapEnd - chunkStart;
+
+ int firstChecksumIndex = Math.toIntExact(offsetInChunk /
bytesPerChecksum);
+ int lastChecksumIndex = Math.toIntExact((endOffsetInChunk - 1) /
bytesPerChecksum);
+
+ checksumIndex += (lastChecksumIndex - firstChecksumIndex + 1);
+ }
+ currentChunkOffset += chunk.getLen();
+ }
+ }
+ }
Review Comment:
> Questions: Is it possible to have chunk length not a multiple of
bytesPerChecksum?
>
> If yes, suppose
>
> bytesPerChecksum is 16, and
> chunkList(offset, length): (0, 10), (10, 20), (30, 10)
> Then, how many checksums does it need? Is it 4 (= 1 + 2 + 1)?
First, yes, it is possible for a chunk's length to not be a multiple of
bytesPerChecksum. When an hsync or manual flush occurs, we cannot guarantee
that the data written by the client will align with a multiple of
bytesPerChecksum.
Second, I would like to clarify how the checksums are calculated.
In your example with chunkList(offset, length): (0, 10), (10, 20), (30, 10),
exactly 3 checksums (1 + 1 + 1) will be generated.
The key point here is how the calculation handles chunks that fall short of
the bytesPerChecksum boundary. When the first chunk isn't a multiple of
bytesPerChecksum (e.g., only 10 bytes), the writer automatically pads the
remaining 6 bytes before computing the checksum. Then, for the next chunk, the
checksum calculation starts entirely fresh from the beginning of that new
chunk, rather than carrying over or continuing from the previous chunk's offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
