Siyao Meng created HDDS-17012:
---------------------------------
Summary: Container data scan cut short by a closed RocksDB is
accepted as complete, so a partial checksum is reported and the replica stays
CLOSED
Key: HDDS-17012
URL: https://issues.apache.org/jira/browse/HDDS-17012
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-3-report-data-scan-cut-short-by-closed-db.patch,
TestBugMC3ScanTruncatedByDbClose.java
h3. Mechanism
Since HDDS-14800, {{RDBStoreAbstractIterator.hasNext}} returns false once the
RocksDB is closed, so a scanner stops without an exception. The datanode block
iterator ({{AbstractDatanodeStore.KeyValueBlockIterator}}) passes that on, so
{{KeyValueContainerCheck.scanData}} takes the end of iteration as the end of
the container's blocks, and a scan whose per-disk DB is closed partway through
ends with no error. {{ContainerScanHelper.scanData}} then treats it as a
complete healthy scan: {{updateContainerChecksum}} writes the merkle tree built
from the blocks scanned so far to the {{.tree}} file, sets the container data
checksum from it and sends an ICR, the data scan timestamp moves forward, and
the replica stays CLOSED.
A closed DB stays closed until the datanode restarts: {{DatanodeStoreCache}}
keeps the closed store, and the volume check does not look at it
({{HddsVolume.checkDbHealth}} also ignores NoSpace errors). The metadata
scanner does not read the DB.
h3. Trigger
# A CLOSED container is being scanned by the background or on demand data
scanner.
# The per-disk RocksDB of its volume is closed by RocksDB error handling
({{RocksDatabase.closeOnError}}), for example when a write or flush for another
container on the same disk fails with an IOError such as a full disk.
# The scanner's next {{hasNext}} returns false and the partial result is saved.
No client action is involved; the close comes from a disk or RocksDB error on
the datanode.
h3. Impact
* Only the container being scanned on that disk when the DB closes is affected
(normally one). Every other container on the disk is already marked UNHEALTHY
with checksum 0 when the scanner or a failed request reaches it, because its
scan finds the DB closed at the start.
* The affected replica stays CLOSED although its DB can no longer be read, and
reports a data checksum computed from part of its blocks. That checksum is
visible to operators ({{ozone admin container info}}) and disagrees with the
other replicas.
* It is marked UNHEALTHY by the next scan of that container. A client request
that needs the DB fails and triggers an on demand scan ({{HddsDispatcher}} does
this for every failed request), but not within {{hdds.container.scrub.min.gap}}
(15 minutes by default) of the truncated scan, since that scan moved the scan
timestamp forward. A container that receives no requests waits for the next
background data scan (the interval is 7 days by default). Argued from the code,
not run end to end.
* No data is lost.
The code is in the released versions 2.2.0 and 2.2.1, which include HDDS-14800.
h3. Reproduction
PASS, deterministic, unmodified source, needs a non root user.
[^TestBugMC3ScanTruncatedByDbClose.java] ({{hadoop-hdds/container-service}})
uses a schema V3 volume, a real {{KeyValueHandler}} and
{{ContainerScanHelper}}. A closed container with 8 blocks is scanned once as a
baseline. A second scan is paused by a throttler at the start of the third
block; meanwhile a block is written for another open container on the same DB,
the DB directory is made read only and the flush fails with an IOError, which
closes the DB through {{closeOnError}}. After the scan resumes the container is
still CLOSED, its checksum changed, the tree file holds 3 of 8 blocks, the last
ICR carries the new checksum with state CLOSED and the scan timestamp moved
forward. A passing test means the defect is present.
h3. Patch
[^MC-3-report-data-scan-cut-short-by-closed-db.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3.
When the underlying table iterator ends, {{KeyValueBlockIterator.hasNext}} now
throws an {{IOException}} if the DB is closed, instead of reporting the end of
the blocks. It reads the unsynchronized {{RDBStore.isClosed}}, so it cannot
block against {{AbstractRDBStore.stop}}, which waits for open iterators. The
shared {{RDBStoreAbstractIterator}} used by OM, SCM and Recon is not changed.
Every caller of the block iterator already handles an {{IOException}} from it,
since creating the iterator on a closed DB already throws one. Besides the
scanner this covers {{KeyValueContainerUtil.noBlocksInContainer}} (the empty
check before a container delete and after block deletion), which could
otherwise answer "no blocks" if the DB closed during its iteration, and the
block counting and metadata checksum paths.
{{KeyValueContainerCheck.scanData}} records the exception as
{{INACCESSIBLE_DB}} and, when the block list could not be read in full, now
returns an empty data tree, as it already does when the DB cannot be opened at
the start of a scan ({{DataScanResult.unhealthyMetadata}} uses the same
convention). So the replica is marked UNHEALTHY with checksum 0, the same as
the other containers on that disk. A scan that finishes on an open DB and a
scan of a container deleted during the scan behave as before.
Covered by {{testKeyValueContainerCheckDBClosedDuringScan}} in the existing
{{TestKeyValueContainerCheck}}, which stops the store from another thread
during the scan and expects one {{INACCESSIBLE_DB}} error and an empty tree
with checksum 0. Without the change it fails for all container test versions
with no scan error reported. With it the block iterator, scanner and
reconciliation suites and the suites of the other block iterator callers (887
tests) and the three container scanner integration suites on a mini cluster (21
tests) pass, and checkstyle is clean. With the patch the reproduction fails
because the replica is marked UNHEALTHY and reports checksum 0.
Not changed here: a per-disk DB closed by RocksDB does not fail its volume, so
after a transient full disk every container on it ends up UNHEALTHY, even
though a restart would reopen the DB. That is a separate follow up (related:
HDDS-13994, opening the DB read only on a full disk).
Found by TLA+ model checking and code review of the datanode volume and
per-disk RocksDB lifecycle paths under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
