Siyao Meng created HDDS-16974:
---------------------------------
Summary: EC file checksum is not bounded by the committed block
group length when a replica lists a stripe beyond it
Key: HDDS-16974
URL: https://issues.apache.org/jira/browse/HDDS-16974
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-1-bound-ec-block-checksum-by-committed-block-length.patch
h3. Mechanism
{{ECBlockChecksumComputer}} computes the checksum of an EC block group from the
stripe checksums in the chunk list of one replica
({{ECFileChecksumHelper.getChunkInfos}}, one {{ChunkInfo}} per stripe). It is
given the block group length committed in OM as {{blockLength}}, but neither
mode stops at that length:
* {{computeMd5Crc}} never reads {{blockLength}} and digests the stripe
checksums of every listed stripe.
* {{computeCompositeCrc}} counts {{blockSize}} down from {{blockLength}}, but
its loop does not end when the length is used up.
A replica can list a stripe beyond the committed length. When the PutBlock of a
stripe fails on one replica, {{ECKeyOutputStream}} writes the stripe again into
a new block group, and the replicas that stored it keep it in their chunk list.
The data read path is bounded by the OM length, the javadoc of
{{ECReconstructionCoordinator.calcEffectiveBlockGroupLen}} accepts the extra
stripe for that reason, and HDDS-10985 bounded the PutBlock of reconstruction
for the same state. The checksum path was not adapted.
h3. Trigger
# A replica of an EC block group lists a stripe beyond the length committed for
that block group. Reached on a mini cluster by stopping one datanode between
two stripes of a key write (rs-3-2-1024k, replica 2, last stripe shorter than
one cell).
# The file checksum of the key is requested ({{ozone sh key checksum}}, or
{{FileSystem#getFileChecksum}} of ofs and o3fs).
h3. Impact
The key is committed and reads back correctly, only its file checksum is
affected.
* MD5MD5CRC: the checksum includes the extra stripe. Two keys with the same
bytes and the same block layout got different checksums (run on the mini
cluster).
* COMPOSITE_CRC, the default of {{ozone.client.checksum.combine.mode}}: the
computation does not stop at the block length. On the mini cluster the call for
the affected key had not returned after 20 seconds, while keys with the same
bytes and no extra stripe returned at once.
* Nothing removes the extra stripe from the replicas (code reading, not run),
so the key behaves this way for as long as the checksum code does not stop at
the block length. Affected keys need no repair: with the patch the affected key
returned the same checksum as the unaffected keys in both modes.
h3. Reproduction
No standalone reproduction is attached to this issue. The regression test in
the patch,
{{TestECBlockChecksumComputer.testStripeBeyondBlockLengthIsIgnored}}, covers
both modes at unit level and fails on unmodified source: the MD5MD5CRC case
with a different digest, the COMPOSITE_CRC case with its 30 second timeout.
h3. Patch
[^MC-1-bound-ec-block-checksum-by-committed-block-length.patch], against
031e76aa2c9a60b67185390b5b33499cbbb82e43. It also applies to master at
c5d4f28266d.
{{ECBlockChecksumComputer}} stops at the block length in both modes.
{{computeMd5Crc}} counts the block length down by one stripe per {{ChunkInfo}}
and skips what lies beyond it. The loop in {{computeCompositeCrc}} ends once
nothing of the block length is left to consume. A block whose chunk list
matches its length gets the same result as before: generated chunk lists over a
range of cell sizes, bytes per checksum and block lengths gave identical
results with and without the patch, and {{TestOzoneFileChecksum}} passes on a
mini cluster (6 tests). The write path is left as it is, so the replicas keep
the extra stripe, as they do today for reads and reconstruction.
With the patch the checksum unit suites of {{hadoop-ozone/client}} pass (27
tests) and checkstyle is clean.
Found by TLA+ model checking and code review of the client EC key write and
file checksum paths under HDDS-15926, on commit
031e76aa2c9a60b67185390b5b33499cbbb82e43. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-10985 (resolved, the same state in reconstruction) and
HDDS-16834 (open, a block without CRC bytes in COMPOSITE_CRC mode). The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
