[
https://issues.apache.org/jira/browse/HDDS-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16974:
------------------------------
Attachment: MC-1-bound-ec-block-checksum-by-committed-block-length.patch
> EC file checksum is not bounded by the committed block group length when a
> replica lists a stripe beyond it
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16974
> URL: https://issues.apache.org/jira/browse/HDDS-16974
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-1-bound-ec-block-checksum-by-committed-block-length.patch
>
>
> h3. Mechanism
> {{ECBlockChecksumComputer}} computes the checksum of an EC block group from
> the stripe checksums in the chunk list of one replica
> ({{ECFileChecksumHelper.getChunkInfos}}, one {{ChunkInfo}} per stripe). It is
> given the block group length committed in OM as {{blockLength}}, but neither
> mode stops at that length:
> * {{computeMd5Crc}} never reads {{blockLength}} and digests the stripe
> checksums of every listed stripe.
> * {{computeCompositeCrc}} counts {{blockSize}} down from {{blockLength}}, but
> its loop does not end when the length is used up.
> A replica can list a stripe beyond the committed length. When the PutBlock of
> a stripe fails on one replica, {{ECKeyOutputStream}} writes the stripe again
> into a new block group, and the replicas that stored it keep it in their
> chunk list. The data read path is bounded by the OM length, the javadoc of
> {{ECReconstructionCoordinator.calcEffectiveBlockGroupLen}} accepts the extra
> stripe for that reason, and HDDS-10985 bounded the PutBlock of reconstruction
> for the same state. The checksum path was not adapted.
> h3. Trigger
> # A replica of an EC block group lists a stripe beyond the length committed
> for that block group. Reached on a mini cluster by stopping one datanode
> between two stripes of a key write (rs-3-2-1024k, replica 2, last stripe
> shorter than one cell).
> # The file checksum of the key is requested ({{ozone sh key checksum}}, or
> {{FileSystem#getFileChecksum}} of ofs and o3fs).
> h3. Impact
> The key is committed and reads back correctly, only its file checksum is
> affected.
> * MD5MD5CRC: the checksum includes the extra stripe. Two keys with the same
> bytes and the same block layout got different checksums (run on the mini
> cluster).
> * COMPOSITE_CRC, the default of {{ozone.client.checksum.combine.mode}}: the
> computation does not stop at the block length. On the mini cluster the call
> for the affected key had not returned after 20 seconds, while keys with the
> same bytes and no extra stripe returned at once.
> * Nothing removes the extra stripe from the replicas (code reading, not run),
> so the key behaves this way for as long as the checksum code does not stop at
> the block length. Affected keys need no repair: with the patch the affected
> key returned the same checksum as the unaffected keys in both modes.
> h3. Reproduction
> No standalone reproduction is attached to this issue. The regression test in
> the patch,
> {{TestECBlockChecksumComputer.testStripeBeyondBlockLengthIsIgnored}}, covers
> both modes at unit level and fails on unmodified source: the MD5MD5CRC case
> with a different digest, the COMPOSITE_CRC case with its 30 second timeout.
> h3. Patch
> [^MC-1-bound-ec-block-checksum-by-committed-block-length.patch], against
> 031e76aa2c9a60b67185390b5b33499cbbb82e43. It also applies to master at
> c5d4f28266d.
> {{ECBlockChecksumComputer}} stops at the block length in both modes.
> {{computeMd5Crc}} counts the block length down by one stripe per
> {{ChunkInfo}} and skips what lies beyond it. The loop in
> {{computeCompositeCrc}} ends once nothing of the block length is left to
> consume. A block whose chunk list matches its length gets the same result as
> before: generated chunk lists over a range of cell sizes, bytes per checksum
> and block lengths gave identical results with and without the patch, and
> {{TestOzoneFileChecksum}} passes on a mini cluster (6 tests). The write path
> is left as it is, so the replicas keep the extra stripe, as they do today for
> reads and reconstruction.
> With the patch the checksum unit suites of {{hadoop-ozone/client}} pass (27
> tests) and checkstyle is clean.
> Found by TLA+ model checking and code review of the client EC key write and
> file checksum paths under HDDS-15926, on commit
> 031e76aa2c9a60b67185390b5b33499cbbb82e43. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-10985 (resolved, the same state in reconstruction) and
> HDDS-16834 (open, a block without CRC bytes in COMPOSITE_CRC mode). The
> attached patch is a proposal for review. Generated with Specula (Claude Opus
> 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
