Siyao Meng created HDDS-16805:
---------------------------------
Summary: Container reconciliation does not verify chunks pulled
from a peer when ozone.client.verify.checksum is false on the datanode
Key: HDDS-16805
URL: https://issues.apache.org/jira/browse/HDDS-16805
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-7-verify-peer-chunk-checksums-in-reconciliation.patch,
TestBugMC7ReconcileUnverifiedPeerChunk.java
h3. Mechanism
{{KeyValueHandler.reconcileChunksPerBlock}} reads chunks from a peer through a
{{BlockInputStream}} that it creates with
{{conf.getObject(OzoneClientConfig.class)}}, the datanode's own view of the
client configuration. The client streams verify what they read only when
{{ozone.client.verify.checksum}} is true, and the peer does not verify on
{{ReadChunk}}. With the setting false in the datanode's configuration, a chunk
that is corrupt on the peer is written into the local replica, the local merkle
tree records it as healthy with the checksum from the peer's block metadata,
and the block is committed with the peer's BCSID. A chunk that the local tree
lists as healthy is never pulled, so this affects blocks and chunks that the
replica lacks or that its own scanner marked corrupt.
The peer's tree is only as fresh as the peer's last scan, so a chunk that went
bad since then is still listed as healthy and offered for repair.
The reconciliation design document says that the datanode validates the
checksum of a chunk read from a peer before using it, and that the existing
client code already does this. That holds only while the client setting is on.
{{ECReconstructionCoordinator}}, the other datanode path that reads peer data
through the client streams, calls {{setChecksumVerify(true)}} on its config
before it opens the stream. The line in {{reconcileChunksPerBlock}} dates from
HDDS-11763 (read from history).
h3. Trigger
# {{ozone.client.verify.checksum=false}} is in the configuration that the
datanode loads, for example a site file shared with clients.
# A replica lacks a block or chunk that a peer has, or holds a chunk that its
own scanner marked corrupt.
# On the peer that chunk is corrupt on disk and the peer has not been scanned
since.
# An administrator runs {{ozone admin container reconcile}} for the container.
h3. Impact
* The corrupt chunk is copied into the reconciling replica and committed. A
later round against a healthy peer in the same reconcile task does not repair
it, because the local tree already lists the chunk as healthy: after
reconciling with the corrupt peer and then a healthy one, the block still holds
the corrupt bytes (run).
* The on demand scan at the end of the reconcile task finds the mismatch and
marks the replica UNHEALTHY. In the reproduction a replica that was CLOSED and
only lacked one block before the reconcile is UNHEALTHY after it (run).
* With the default setting the corrupt chunk is rejected with a checksum
mismatch and the block is repaired from the healthy peer (run, as the control).
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC7ReconcileUnverifiedPeerChunk.java]
({{hadoop-hdds/container-service}}) uses the setup of
{{TestContainerReconciliationWithMockDatanodes}}: three datanodes, each a real
{{KeyValueHandler}} with a real container on disk and a real on demand scanner,
the real {{BlockInputStream}}, and only the static {{ContainerProtocolCalls}}
transport routed to the peer's handler. There is no SCM. One replica loses a
block, one peer has a byte flipped in that block and is not scanned, the other
peer is healthy. The replica reconciles with the corrupt peer first and the
healthy peer second. With the setting false on the reconciling datanode the
block is committed with the peer's BCSID, the block file equals the corrupt
peer's bytes, and the replica is UNHEALTHY after the scan. With the setting
true the block file equals the healthy bytes and the replica stays CLOSED. A
passing test means the defect is present.
h3. Patch
[^MC-7-verify-peer-chunk-checksums-in-reconciliation.patch], against
1e528b97aed3eef1be6846f1f0767c1ef98a883a. It also applies to master at
8ca44914771.
{{reconcileChunksPerBlock}} calls {{setChecksumVerify(true)}} on the
{{OzoneClientConfig}} that it passes to the stream, as
{{ECReconstructionCoordinator}} does. {{getObject}} returns a new object on
every call, so no other code sees the flag. Nothing changes with the default
setting. With the setting false, reconciliation now computes checksums over
what it pulls, and a chunk that fails ends the repair of that block from that
peer, which is the existing handling of a chunk that cannot be pulled. The next
peer is then tried.
Covered by
{{testContainerReconciliationVerifiesPeerDataWhenClientChecksumVerifyDisabled}}
in the existing {{TestContainerReconciliationWithMockDatanodes}}. Without the
change it fails because the data checksum of the replica after reconcile and
scan is not the healthy one. With it the unit suites of the {{keyvalue}},
{{ozoneimpl}} and {{checksum}} packages in {{hadoop-hdds/container-service}}
pass (1124 tests, 18 skipped by the suites), checkstyle is clean, and the case
of the reproduction with the setting false fails because the block now holds
the healthy bytes. Not run on a mini cluster.
Found by TLA+ model checking and code review of the container reconciliation
paths in the datanode and SCM under HDDS-15926, on commit
1e528b97aed3eef1be6846f1f0767c1ef98a883a. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
