[
https://issues.apache.org/jira/browse/HDDS-16873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16873:
-------------------------------------
Assignee: Andrey Yarovoy
> Report replicas that fail client-side checksum verification
> -----------------------------------------------------------
>
> Key: HDDS-16873
> URL: https://issues.apache.org/jira/browse/HDDS-16873
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h2. Problem
> The datanode does not verify chunk checksums for client reads.
> {\{KeyValueHandler}} validates them only for Ratis-internal reads; for client
> reads "the client is expected to validate". When the client finds a mismatch
> (\{{OzoneChecksumException}}), it reads another replica (Ratis) or treats the
> location as failed and reconstructs (EC), and the read succeeds. Nobody is
> told:
> * the read returned \{{SUCCESS}} on the datanode, so \{{HddsDispatcher}} does
> not
> trigger an on-demand container scan (it does so only for non-\{{SUCCESS}}
> results);
> * SCM is not informed.
> The corrupt replica stays in service until the background data scanner reaches
> it. Until then every reader of that data pays the retry or the reconstruction,
> and redundancy is lower than SCM believes.
> h2. HDFS behavior
> {\{DFSInputStream}} / \{{DFSStripedInputStream}} collect checksum failures in
> {\{CorruptedBlocks}} and call \{{reportCheckSumFailure}} in \{{finally}},
> which
> reports the replica to the NameNode; the NameNode marks it corrupt and
> re-replicates or reconstructs it.
> h2. Proposal
> Client asks the datanode to scan.* Add a container
> request with which the client reports a checksum failure for a block on that
> datanode. The datanode does not trust the report: it runs an on-demand scan of
> the container (\{{OnDemandContainerScanner}}), rate-limited by the scanner's
> existing per-container minimum gap. If the scan confirms the corruption, the
> container is marked unhealthy and the existing ICR -> SCM ReplicationManager
> path repairs it. The client sends the report asynchronously, at most once per
> datanode and container per stream, and it never fails or delays the read.
> Security: the request must be authorized like a read of the block (block /
> container token in secure mode), so a client can only trigger scans on data it
> can read, and the scanner's rate limit bounds the load it can cause. Assess it
> against \{{THREAT_MODEL.md}} before implementation.
> h2. Testing
> * Unit: a checksum failure on read sends one report per datanode and
> container,
> asynchronously; the read still returns correct data; no report goes to a
> datanode version without support.
> * Datanode unit: the request triggers an on-demand scan and is rate-limited by
> the scanner gap; an unauthorized request is rejected in secure mode.
> * Integration, extending the existing container scanner integration tests
> (\{{TestOnDemandContainerScannerIntegration}}): corrupt a chunk on one
> replica,
> read the key, and check that the container becomes unhealthy on that datanode
> and is repaired; cover a Ratis key and an EC key.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
