ptlrs commented on code in PR #9947:
URL: https://github.com/apache/ozone/pull/9947#discussion_r2984253465
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java:
##########
@@ -326,17 +327,30 @@ public VolumeCheckResult checkDbHealth(File dbFile)
throws InterruptedException
return VolumeCheckResult.HEALTHY;
}
+ // We attempt to open RocksDb twice to ignore any transient errors
+ // and to confirm that we actually cannot open RocksDb in readonly mode.
final boolean isVolumeTestResultHealthy = true;
- try (ManagedOptions managedOptions = new ManagedOptions();
- ManagedRocksDB ignored = ManagedRocksDB.openReadOnly(managedOptions,
dbFile.toString())) {
- volumeTestResultQueue.add(isVolumeTestResultHealthy);
- } catch (Exception e) {
- if (Thread.currentThread().isInterrupted()) {
- throw new InterruptedException("Check of database for volume " + this
+ " interrupted.");
+ final int maxAttempts = 2;
+ final Duration maxRetryGap = getDatanodeConfig().getDiskCheckRetryGap();
+ for (int attempt = 0; attempt < maxAttempts; attempt++) {
+ try (ManagedOptions managedOptions = new ManagedOptions();
+ ManagedRocksDB ignored =
ManagedRocksDB.openReadOnly(managedOptions, dbFile.toString())) {
+ volumeTestResultQueue.add(isVolumeTestResultHealthy);
+ break;
+ } catch (Exception e) {
+ if (Thread.currentThread().isInterrupted()) {
+ throw new InterruptedException("Check of database for volume " +
this + " interrupted.");
+ }
+
+ if (attempt == maxAttempts - 1) {
Review Comment:
Yes, the idea is that if you can't open the RocksDb twice, the problem
should definitely be counted as an error.
There is no clear list of the possible transient errors, as such this check
is purely defensive.
From the RocksDb docs and comments, it appears that we may not be able to
open the database in a Read-Only mode when any of these happen:
1. CURRENT file is being renamed
2. SST files are deleted or some states of compaction
3. WAL physical size on disk exceeds declared size in MANIFEST
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
