Siyao Meng created HDDS-16806:
---------------------------------
Summary: Container scanner builds an empty merkle tree for
containers written with checksum type NONE, which breaks reconciliation of
those containers
Key: HDDS-16806
URL: https://issues.apache.org/jira/browse/HDDS-16806
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-4-add-chunks-without-checksums-to-scanner-merkle-tree.patch,
TestBugCR4ScannerTreeWithoutChecksums.java
h3. Mechanism
The merkle tree of a container replica is built in two places.
{{KeyValueHandler.updateAndGetContainerChecksumFromMetadata}} adds every block
and chunk found in the block metadata. {{KeyValueContainerCheck.scanBlock}},
used by the data scan, adds a block and its chunks only inside the branch for
chunks whose checksum type is not {{NONE}}. For a container written with
{{ozone.client.checksum.type=NONE}} a data scan therefore replaces the tree
with one that has no live blocks (data checksum 0 if no block deletion has been
recorded). The replica itself stays healthy.
Reconciliation compares trees, so the empty tree has two effects:
* A scanned replica sees every block of a peer that still has the tree from
metadata as missing, pulls all of them and overwrites its own block files with
the peer's bytes.
* Two scanned replicas both have trees without live blocks, so reconciliation
finds no difference between them whatever live blocks they hold.
For CRC32 the two builders agree, and
{{TestContainerReconciliationWithMockDatanodes.setup}} asserts that. No
existing test covers {{NONE}}.
The scanner has built its tree this way since HDDS-10374. Blocks that are
present locally are pulled again since HDDS-14763, which removed the check that
skipped them (both read from history, not run on a release).
h3. Trigger
# Blocks are written with {{ozone.client.checksum.type=NONE}} and the container
is closed. All replicas get a tree from metadata that lists every block.
# The data scanner (background or on demand) scans one replica. Its tree is now
empty. An on demand scan also runs automatically when a replica is imported by
container replication, and the tree file travels with the container, so a
freshly replicated replica has the empty tree while its source still has the
tree from metadata, without any scan timing involved (read from code, not run).
# An administrator runs {{ozone admin container reconcile}}. The scanned
replica pulls every block from a peer that has not been scanned yet.
For the second effect: all replicas have been scanned, one of them has lost a
block, and the reconcile command does not restore it. The scan at the end of
every reconcile task empties the tree of the replica that ran it, so a
reconcile of such a container moves its replicas into this state (read from
code).
h3. Impact
* Present, healthy block files are replaced by the peer's copy, which cannot be
verified for type {{NONE}}. If the peer's copy is damaged, the good copy is
lost. In the reproduction the peer has one flipped byte and the scanned replica
ends up with it (run).
* A block that a replica has lost is not repaired by reconciliation once the
replicas are scanned, and all replicas report the same data checksum (0 if
nothing was deleted), so the divergence is not visible in the reported
checksums either (run at the datanode, SCM not run).
* Only containers written with checksum type {{NONE}} are affected.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugCR4ScannerTreeWithoutChecksums.java]
({{hadoop-hdds/container-service}}) uses the setup of
{{TestContainerReconciliationWithMockDatanodes}}: two datanodes, each a real
{{KeyValueHandler}} with a real container on disk and a real on demand scanner,
the real {{BlockInputStream}}, and only the static {{ContainerProtocolCalls}}
transport routed to the peer's handler. There is no SCM. Three tests: the
scanned tree of a {{NONE}} container is empty with data checksum 0 while the
replica stays CLOSED (CRC32 as the control); a scanned replica is overwritten
with the peer's flipped byte copy, and is left alone when it was not scanned
(control); a lost block is still absent after reconcile when both replicas were
scanned. A passing test means the defect is present.
h3. Patch
[^CR-4-add-chunks-without-checksums-to-scanner-merkle-tree.patch], against
1e528b97aed3eef1be6846f1f0767c1ef98a883a. It also applies to master at
8ca44914771.
{{scanBlock}} adds a chunk without checksums to its tree as healthy, with the
same {{addChunks}} call that the metadata builder uses, so both builders
produce the same tree for a healthy {{NONE}} container. Blocks without chunk
data on disk are still listed only by the metadata builder, for every checksum
type; that difference is older and is not changed here. Nothing changes for
containers with a real checksum type. For a {{NONE}} container that was already
scanned, the reported data checksum changes from 0 to the metadata value at its
next data scan, and until all its replicas have been rescanned a rescanned
replica and one that was not differ as a scanned and an unscanned one do today.
Containers whose replicas had all been scanned already, where reconcile did
nothing, get that window again for up to one data scan cycle after the upgrade,
and Recon shows them as REPLICA_MISMATCH during it. Nothing reconciles
automatically. The patch does not make corruption inside a {{NONE}} chunk
detectable: the scanner still does not read such a chunk and does not check its
length.
Covered by {{testContainerWithoutChunkChecksumsAfterScan}} in the existing
{{TestContainerReconciliationWithMockDatanodes}}, which checks that the scanned
data checksum equals the one from metadata, that a replica with a lost block is
visible after all replicas are scanned, and that reconciliation repairs it.
Without the change it fails on the first check (data checksum 0). With it the
unit suites of the {{keyvalue}}, {{ozoneimpl}} and {{checksum}} packages in
{{hadoop-hdds/container-service}} pass (1124 tests, 18 skipped by the suites,
including the scanner suites), checkstyle is clean, and the three bug cases of
the reproduction fail because the scanned tree is no longer empty. Not run on a
mini cluster.
Found by code review of the container reconciliation paths in the datanode and
SCM, as part of the TLA+ verification effort under HDDS-15926, on commit
1e528b97aed3eef1be6846f1f0767c1ef98a883a. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
