[
https://issues.apache.org/jira/browse/HDDS-16806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16806:
------------------------------
Attachment: CR-4-add-chunks-without-checksums-to-scanner-merkle-tree.patch
> Container scanner builds an empty merkle tree for containers written with
> checksum type NONE, which breaks reconciliation of those containers
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16806
> URL: https://issues.apache.org/jira/browse/HDDS-16806
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-4-add-chunks-without-checksums-to-scanner-merkle-tree.patch,
> TestBugCR4ScannerTreeWithoutChecksums.java
>
>
> h3. Mechanism
> The merkle tree of a container replica is built in two places.
> {{KeyValueHandler.updateAndGetContainerChecksumFromMetadata}} adds every
> block and chunk found in the block metadata.
> {{KeyValueContainerCheck.scanBlock}}, used by the data scan, adds a block and
> its chunks only inside the branch for chunks whose checksum type is not
> {{NONE}}. For a container written with {{ozone.client.checksum.type=NONE}} a
> data scan therefore replaces the tree with one that has no live blocks (data
> checksum 0 if no block deletion has been recorded). The replica itself stays
> healthy.
> Reconciliation compares trees, so the empty tree has two effects:
> * A scanned replica sees every block of a peer that still has the tree from
> metadata as missing, pulls all of them and overwrites its own block files
> with the peer's bytes.
> * Two scanned replicas both have trees without live blocks, so reconciliation
> finds no difference between them whatever live blocks they hold.
> For CRC32 the two builders agree, and
> {{TestContainerReconciliationWithMockDatanodes.setup}} asserts that. No
> existing test covers {{NONE}}.
> The scanner has built its tree this way since HDDS-10374. Blocks that are
> present locally are pulled again since HDDS-14763, which removed the check
> that skipped them (both read from history, not run on a release).
> h3. Trigger
> # Blocks are written with {{ozone.client.checksum.type=NONE}} and the
> container is closed. All replicas get a tree from metadata that lists every
> block.
> # The data scanner (background or on demand) scans one replica. Its tree is
> now empty. An on demand scan also runs automatically when a replica is
> imported by container replication, and the tree file travels with the
> container, so a freshly replicated replica has the empty tree while its
> source still has the tree from metadata, without any scan timing involved
> (read from code, not run).
> # An administrator runs {{ozone admin container reconcile}}. The scanned
> replica pulls every block from a peer that has not been scanned yet.
> For the second effect: all replicas have been scanned, one of them has lost a
> block, and the reconcile command does not restore it. The scan at the end of
> every reconcile task empties the tree of the replica that ran it, so a
> reconcile of such a container moves its replicas into this state (read from
> code).
> h3. Impact
> * Present, healthy block files are replaced by the peer's copy, which cannot
> be verified for type {{NONE}}. If the peer's copy is damaged, the good copy
> is lost. In the reproduction the peer has one flipped byte and the scanned
> replica ends up with it (run).
> * A block that a replica has lost is not repaired by reconciliation once the
> replicas are scanned, and all replicas report the same data checksum (0 if
> nothing was deleted), so the divergence is not visible in the reported
> checksums either (run at the datanode, SCM not run).
> * Only containers written with checksum type {{NONE}} are affected.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugCR4ScannerTreeWithoutChecksums.java]
> ({{hadoop-hdds/container-service}}) uses the setup of
> {{TestContainerReconciliationWithMockDatanodes}}: two datanodes, each a real
> {{KeyValueHandler}} with a real container on disk and a real on demand
> scanner, the real {{BlockInputStream}}, and only the static
> {{ContainerProtocolCalls}} transport routed to the peer's handler. There is
> no SCM. Three tests: the scanned tree of a {{NONE}} container is empty with
> data checksum 0 while the replica stays CLOSED (CRC32 as the control); a
> scanned replica is overwritten with the peer's flipped byte copy, and is left
> alone when it was not scanned (control); a lost block is still absent after
> reconcile when both replicas were scanned. A passing test means the defect is
> present.
> h3. Patch
> [^CR-4-add-chunks-without-checksums-to-scanner-merkle-tree.patch], against
> 1e528b97aed3eef1be6846f1f0767c1ef98a883a. It also applies to master at
> 8ca44914771.
> {{scanBlock}} adds a chunk without checksums to its tree as healthy, with the
> same {{addChunks}} call that the metadata builder uses, so both builders
> produce the same tree for a healthy {{NONE}} container. Blocks without chunk
> data on disk are still listed only by the metadata builder, for every
> checksum type; that difference is older and is not changed here. Nothing
> changes for containers with a real checksum type. For a {{NONE}} container
> that was already scanned, the reported data checksum changes from 0 to the
> metadata value at its next data scan, and until all its replicas have been
> rescanned a rescanned replica and one that was not differ as a scanned and an
> unscanned one do today. Containers whose replicas had all been scanned
> already, where reconcile did nothing, get that window again for up to one
> data scan cycle after the upgrade, and Recon shows them as REPLICA_MISMATCH
> during it. Nothing reconciles automatically. The patch does not make
> corruption inside a {{NONE}} chunk detectable: the scanner still does not
> read such a chunk and does not check its length.
> Covered by {{testContainerWithoutChunkChecksumsAfterScan}} in the existing
> {{TestContainerReconciliationWithMockDatanodes}}, which checks that the
> scanned data checksum equals the one from metadata, that a replica with a
> lost block is visible after all replicas are scanned, and that reconciliation
> repairs it. Without the change it fails on the first check (data checksum 0).
> With it the unit suites of the {{keyvalue}}, {{ozoneimpl}} and {{checksum}}
> packages in {{hadoop-hdds/container-service}} pass (1124 tests, 18 skipped by
> the suites, including the scanner suites), checkstyle is clean, and the three
> bug cases of the reproduction fail because the scanned tree is no longer
> empty. Not run on a mini cluster.
> Found by code review of the container reconciliation paths in the datanode
> and SCM, as part of the TLA+ verification effort under HDDS-15926, on commit
> 1e528b97aed3eef1be6846f1f0767c1ef98a883a. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
