[
https://issues.apache.org/jira/browse/HDDS-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17012:
------------------------------
Attachment: TestBugMC3ScanTruncatedByDbClose.java
> Container data scan cut short by a closed RocksDB is accepted as complete, so
> a partial checksum is reported and the replica stays CLOSED
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17012
> URL: https://issues.apache.org/jira/browse/HDDS-17012
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-3-report-data-scan-cut-short-by-closed-db.patch,
> TestBugMC3ScanTruncatedByDbClose.java
>
>
> h3. Mechanism
> Since HDDS-14800, {{RDBStoreAbstractIterator.hasNext}} returns false once the
> RocksDB is closed, so a scanner stops without an exception. The datanode
> block iterator ({{AbstractDatanodeStore.KeyValueBlockIterator}}) passes that
> on, so {{KeyValueContainerCheck.scanData}} takes the end of iteration as the
> end of the container's blocks, and a scan whose per-disk DB is closed partway
> through ends with no error. {{ContainerScanHelper.scanData}} then treats it
> as a complete healthy scan: {{updateContainerChecksum}} writes the merkle
> tree built from the blocks scanned so far to the {{.tree}} file, sets the
> container data checksum from it and sends an ICR, the data scan timestamp
> moves forward, and the replica stays CLOSED.
> A closed DB stays closed until the datanode restarts: {{DatanodeStoreCache}}
> keeps the closed store, and the volume check does not look at it
> ({{HddsVolume.checkDbHealth}} also ignores NoSpace errors). The metadata
> scanner does not read the DB.
> h3. Trigger
> # A CLOSED container is being scanned by the background or on demand data
> scanner.
> # The per-disk RocksDB of its volume is closed by RocksDB error handling
> ({{RocksDatabase.closeOnError}}), for example when a write or flush for
> another container on the same disk fails with an IOError such as a full disk.
> # The scanner's next {{hasNext}} returns false and the partial result is
> saved.
> No client action is involved; the close comes from a disk or RocksDB error on
> the datanode.
> h3. Impact
> * Only the container being scanned on that disk when the DB closes is
> affected (normally one). Every other container on the disk is already marked
> UNHEALTHY with checksum 0 when the scanner or a failed request reaches it,
> because its scan finds the DB closed at the start.
> * The affected replica stays CLOSED although its DB can no longer be read,
> and reports a data checksum computed from part of its blocks. That checksum
> is visible to operators ({{ozone admin container info}}) and disagrees with
> the other replicas.
> * It is marked UNHEALTHY by the next scan of that container. A client request
> that needs the DB fails and triggers an on demand scan ({{HddsDispatcher}}
> does this for every failed request), but not within
> {{hdds.container.scrub.min.gap}} (15 minutes by default) of the truncated
> scan, since that scan moved the scan timestamp forward. A container that
> receives no requests waits for the next background data scan (the interval is
> 7 days by default). Argued from the code, not run end to end.
> * No data is lost.
> The code is in the released versions 2.2.0 and 2.2.1, which include
> HDDS-14800.
> h3. Reproduction
> PASS, deterministic, unmodified source, needs a non root user.
> [^TestBugMC3ScanTruncatedByDbClose.java] ({{hadoop-hdds/container-service}})
> uses a schema V3 volume, a real {{KeyValueHandler}} and
> {{ContainerScanHelper}}. A closed container with 8 blocks is scanned once as
> a baseline. A second scan is paused by a throttler at the start of the third
> block; meanwhile a block is written for another open container on the same
> DB, the DB directory is made read only and the flush fails with an IOError,
> which closes the DB through {{closeOnError}}. After the scan resumes the
> container is still CLOSED, its checksum changed, the tree file holds 3 of 8
> blocks, the last ICR carries the new checksum with state CLOSED and the scan
> timestamp moved forward. A passing test means the defect is present.
> h3. Patch
> [^MC-3-report-data-scan-cut-short-by-closed-db.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3.
> When the underlying table iterator ends, {{KeyValueBlockIterator.hasNext}}
> now throws an {{IOException}} if the DB is closed, instead of reporting the
> end of the blocks. It reads the unsynchronized {{RDBStore.isClosed}}, so it
> cannot block against {{AbstractRDBStore.stop}}, which waits for open
> iterators. The shared {{RDBStoreAbstractIterator}} used by OM, SCM and Recon
> is not changed. Every caller of the block iterator already handles an
> {{IOException}} from it, since creating the iterator on a closed DB already
> throws one. Besides the scanner this covers
> {{KeyValueContainerUtil.noBlocksInContainer}} (the empty check before a
> container delete and after block deletion), which could otherwise answer "no
> blocks" if the DB closed during its iteration, and the block counting and
> metadata checksum paths.
> {{KeyValueContainerCheck.scanData}} records the exception as
> {{INACCESSIBLE_DB}} and, when the block list could not be read in full, now
> returns an empty data tree, as it already does when the DB cannot be opened
> at the start of a scan ({{DataScanResult.unhealthyMetadata}} uses the same
> convention). So the replica is marked UNHEALTHY with checksum 0, the same as
> the other containers on that disk. A scan that finishes on an open DB and a
> scan of a container deleted during the scan behave as before.
> Covered by {{testKeyValueContainerCheckDBClosedDuringScan}} in the existing
> {{TestKeyValueContainerCheck}}, which stops the store from another thread
> during the scan and expects one {{INACCESSIBLE_DB}} error and an empty tree
> with checksum 0. Without the change it fails for all container test versions
> with no scan error reported. With it the block iterator, scanner and
> reconciliation suites and the suites of the other block iterator callers (887
> tests) and the three container scanner integration suites on a mini cluster
> (21 tests) pass, and checkstyle is clean. With the patch the reproduction
> fails because the replica is marked UNHEALTHY and reports checksum 0.
> Not changed here: a per-disk DB closed by RocksDB does not fail its volume,
> so after a transient full disk every container on it ends up UNHEALTHY, even
> though a restart would reopen the DB. That is a separate follow up (related:
> HDDS-13994, opening the DB read only on a full disk).
> Found by TLA+ model checking and code review of the datanode volume and
> per-disk RocksDB lifecycle paths under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
