[
https://issues.apache.org/jira/browse/HDDS-17019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17019:
------------------------------
Attachment: TestBugMC4DbVolumeFailureNotPropagated.java
> Failure of a datanode DB volume is not propagated to the data volumes whose
> RocksDB it holds, so they stay in service
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17019
> URL: https://issues.apache.org/jira/browse/HDDS-17019
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-4-fail-data-volumes-of-failed-db-volume.patch,
> TestBugMC4DbVolumeFailureNotPropagated.java
>
>
> h3. Mechanism
> With {{hdds.datanode.container.db.dir}} set, the schema V3 RocksDB of each
> data volume lives on a separate DB volume. {{OzoneContainer}} registers a
> failed volume listener only on the data volume set; the DB volume set has
> none. When the periodic check fails a DB volume, {{DbVolume.failVolume}}
> closes and uncaches the DB stores below it, but the data volumes that use it
> are left NORMAL: they keep their containers, are reported to SCM as healthy
> and are still chosen for new containers. {{DatanodeStoreCache.getDB}} reopens
> a store that is not cached on its next access, so those data volumes go on
> using a RocksDB on the failed DB volume.
> {{HddsVolume.check}} only checks that the data volume's own RocksDB files can
> be read and opened. It does not look at the DB volume's state, so a DB volume
> that fails its own check while those files stay readable (its root directory
> no longer writable, as reproduced here, or a file system remounted read only
> after errors) leaves the data volumes in service. A DB disk that becomes
> unreadable is already caught by each data volume's own check.
> h3. Trigger
> # A datanode with two or more DB volumes. With a single DB volume the
> datanode already shuts down when it fails, because the default
> {{hdds.datanode.failed.db.volumes.tolerated}} of -1 requires one healthy DB
> volume.
> # A DB volume fails its periodic disk check while the RocksDB files on it
> stay readable.
> No client action is involved.
> h3. Impact
> * The data volumes whose DB is on the failed DB volume stay in service until
> the datanode restarts. SCM is not told and their containers are not
> re-replicated.
> * New containers are placed on those data volumes and their block metadata is
> written to the RocksDB that the cache reopened on the failed DB volume. In
> the reproduction the DB files themselves are still writable, so those writes
> succeed. On a file system remounted read only the reopen or the writes would
> fail instead (argued from the code, not run).
> h3. Reproduction
> PASS, deterministic, unmodified source, needs a POSIX file system and a non
> root user. [^TestBugMC4DbVolumeFailureNotPropagated.java]
> ({{hadoop-hdds/container-service}}) starts a real {{OzoneContainer}} with two
> data volumes and two DB volumes; one data volume has its RocksDB on a DB
> volume, the other on itself. The root directory of that DB volume is made
> read only and the real {{StorageVolumeChecker}} fails it through
> {{MutableVolumeSet.checkAllVolumes}}. The data volume set check that follows
> passes, because the RocksDB files stay readable. The data volume stays NORMAL
> with its DB loaded, a new container is placed on it with the round robin
> policy, its block is committed to a DB file under the failed DB volume, and
> the node report shows the data volume as not failed. A passing test means the
> defect is present.
> h3. Patch
> [^MC-4-fail-data-volumes-of-failed-db-volume.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3.
> {{OzoneContainer}} registers a failed volume listener on the DB volume set.
> It collects the active data volumes whose DB volume has failed and passes
> them to {{MutableVolumeSet.handleVolumeFailures}} of the data volume set
> (made public for this), which is the existing path for a failed data volume:
> it marks them failed, closes their DBs, applies
> {{hdds.datanode.failed.data.volumes.tolerated}} and runs the data set
> listener that removes their containers and sends a full container report. The
> listener runs after the DB set has released its lock, so no locks are nested.
> Checking the DB volume from {{HddsVolume.check}} instead would only act in
> the next periodic check, because data volumes are checked before DB volumes
> in each cycle.
> This changes behaviour on purpose, on nodes with two or more DB volumes:
> losing a DB disk now fails every data volume whose DB is on it, their
> containers are re-replicated, and if that exceeds the tolerated number of
> failed data volumes the datanode shuts down, as for any other data volume
> loss.
> Not covered: a DB volume that fails {{StorageVolumeUtil.checkVolume}} during
> the SCM version handshake at startup. {{VersionEndpointTask}} fails it
> directly through {{MutableVolumeSet.failVolume}} while holding the DB set
> lock, so the listener does not run there, and the data volumes whose DB was
> already loaded on it stay in service. Covering it would need a separate check
> in that task.
> Covered by {{testDbVolumeFailureFailsDataVolumesWithDbOnIt}} in the existing
> {{TestOzoneContainer}}, with three data volumes whose RocksDB is on the
> failing DB volume, on the other DB volume and on the data volume itself.
> Without the change it fails because no data volume is failed; it also fails
> if every data volume with a DB volume is failed. With it the
> {{TestOzoneContainer}}, volume, DB volume, volume checker and volume health
> check suites (138 tests) and {{TestOzoneContainerWithTLS}}, an integration
> test that starts a real {{OzoneContainer}} with a DB volume configured, pass,
> and checkstyle is clean. With the patch the reproduction fails because the
> data volume is FAILED.
> Found by TLA+ model checking and code review of the datanode volume and
> per-disk RocksDB lifecycle paths under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-6616 and HDDS-6541, which added the DB volume check and the
> separate DB location. The attached patch is a proposal for review. Generated
> with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
