jojochuang commented on code in PR #10185:
URL: https://github.com/apache/ozone/pull/10185#discussion_r3573255466
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -4110,6 +4113,16 @@ public synchronized TermIndex
installSnapshotFromLeader(String leaderId) throws
omDBCheckpoint = omRatisSnapshotProvider.
downloadDBSnapshotFromLeader(leaderId);
} catch (IOException ex) {
+ if (OmRatisSnapshotProvider.isDiskFullOrQuotaIOException(ex)) {
+ LOG.error(
+ "Failed to download snapshot from leader {}: local disk appears
full or over quota "
+ + "on the OM ratis snapshot volume (see previous ERROR for
path/usable space). "
+ + "Free disk or adjust {}, {}, or {} before bootstrap can
succeed.",
+ leaderId,
+ OZONE_OM_BOOTSTRAP_MIN_SPACE_KEY,
+ OZONE_OM_BOOTSTRAP_CHECKPOINT_HEADROOM_RATIO_KEY,
+ OZONE_OM_CHECKPOINT_ESTIMATED_SST_BYTES_HEADER);
+ }
Review Comment:
Fixed: provider keeps the detailed disk-full ERROR; OzoneManager now skips
the generic "Failed to download snapshot" log when isDiskFullOrQuotaIOException
is true.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis_snapshot/OmRatisSnapshotProvider.java:
##########
@@ -86,6 +96,88 @@ public class OmRatisSnapshotProvider extends
RDBSnapshotProvider {
private final boolean spnegoEnabled;
private final URLConnectionFactory connectionFactory;
private final boolean useV2CheckpointApi;
+ /** Minimum usable bytes on snapshot volume before download; 0 = disabled. */
+ private final long bootstrapMinSpaceBytes;
+ /** Applied to leader-reported SST byte estimate to reserve tar/unpack
headroom. */
+ private final double bootstrapCheckpointHeadroomRatio;
+
+ private static final class BootstrapSpaceRequirement {
+ private final long requiredBytes;
+ private final boolean usedLeaderEstimateHeader;
+
+ private BootstrapSpaceRequirement(long requiredBytes, boolean
usedLeaderEstimateHeader) {
+ this.requiredBytes = requiredBytes;
+ this.usedLeaderEstimateHeader = usedLeaderEstimateHeader;
+ }
+ }
+
+ /**
+ * Whether this {@link IOException} (or its causes) typically means the
+ * local filesystem ran out of space or hit a quota while writing.
+ */
+ public static boolean isDiskFullOrQuotaIOException(IOException ioe) {
+ for (Throwable t = ioe; t != null; t = t.getCause()) {
+ if (t instanceof FileSystemException) {
+ FileSystemException fse = (FileSystemException) t;
+ String reason = fse.getReason();
+ if (reason != null) {
+ String r = reason.toLowerCase(Locale.ROOT);
+ if (r.contains("no space") || r.contains("space left")
+ || r.contains("quota") || r.contains("enospc")) {
+ return true;
+ }
+ }
+ }
+ String msg = t.getMessage();
+ if (msg != null) {
+ String m = msg.toLowerCase(Locale.ROOT);
+ if (m.contains("no space left on device")
+ || m.contains("enospc")
+ || m.contains("disk quota exceeded")
+ || m.contains("quota exceeded")) {
+ return true;
Review Comment:
Added DiskOutOfSpaceException type check in the cause chain. English
substring matching remains as a supplement for JDK FileSystemException ENOSPC
on Linux. Documented limitation in Javadoc + unit test.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java:
##########
@@ -1005,6 +1015,51 @@ public static Map<String, List<String>>
readFileToMap(String filePath) throws IO
return dataMap;
}
+ /**
+ * Follower bootstrap must abort before streaming when the leader's SST
estimate header
+ * implies more free space than is available (v2 inode-based checkpoint URL).
+ */
+ @Test
+ public void
testBootstrapSnapshotDownloadAbortsWhenDiskSpaceBelowLeaderSstEstimateV2()
Review Comment:
Covered by
testBootstrapSnapshotDownloadAbortsWhenDiskSpaceBelowLeaderSstEstimate with
@ValueSource(booleans = {true, false}) (inode-based and legacy checkpoint URL).
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -2378,6 +2379,16 @@
request OM snapshot from OM Leader.
</description>
</property>
+ <property>
+ <name>ozone.om.bootstrap.min.space</name>
+ <value>5GB</value>
Review Comment:
Leader sends X-Ozone-Om-Checkpoint-Estimated-Sst-Bytes × headroom ratio
(default 2.0). ozone.om.bootstrap.min.space (5GB) is only the fallback when the
header is absent.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java:
##########
@@ -291,6 +291,18 @@ public final class OMConfigKeys {
OZONE_OM_SNAPSHOT_PROVIDER_REQUEST_TIMEOUT_DEFAULT =
TimeDuration.valueOf(300000, TimeUnit.MILLISECONDS);
+ public static final String OZONE_OM_BOOTSTRAP_MIN_SPACE_KEY =
+ "ozone.om.bootstrap.min.space";
+ public static final String OZONE_OM_BOOTSTRAP_MIN_SPACE_DEFAULT = "5GB";
+
+ /**
+ * Multiplier applied to the leader-reported estimated SST bytes when
deciding
+ * minimum free space before downloading a checkpoint (tar + unpack
headroom).
+ */
+ public static final String OZONE_OM_BOOTSTRAP_CHECKPOINT_HEADROOM_RATIO_KEY =
+ "ozone.om.bootstrap.checkpoint.estimated.space.headroom.ratio";
+ public static final double
OZONE_OM_BOOTSTRAP_CHECKPOINT_HEADROOM_RATIO_DEFAULT = 1.25D;
Review Comment:
Done — default headroom ratio is 2.0 in this PR.
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis_snapshot/TestOmRatisSnapshotProvider.java:
##########
@@ -80,6 +87,46 @@ public void setup(@TempDir File snapshotDir,
false, connectionFactory);
}
+ @Test
+ public void testIsDiskFullOrQuotaIOExceptionDetectsNoSpaceMessage() {
+ assertTrue(OmRatisSnapshotProvider.isDiskFullOrQuotaIOException(
+ new IOException("No space left on device")));
+ }
+
+ @Test
+ public void
testIsDiskFullOrQuotaIOExceptionDetectsFileSystemExceptionReason() {
+ IOException wrapped = new IOException("write failed",
+ new FileSystemException("p", null, "No space left on device"));
+ assertTrue(OmRatisSnapshotProvider.isDiskFullOrQuotaIOException(wrapped));
+ }
+
+ @Test
+ public void testIsDiskFullOrQuotaIOExceptionReturnsFalseForOtherErrors() {
+ assertFalse(OmRatisSnapshotProvider.isDiskFullOrQuotaIOException(
+ new IOException("Connection reset")));
+ }
+
+ @Test
+ public void testBootstrapDiskSpaceCheckSkippedWhenZero(@TempDir File
snapshotDir) {
+ OzoneConfiguration conf = new OzoneConfiguration();
+ conf.set(OMConfigKeys.OZONE_OM_BOOTSTRAP_MIN_SPACE_KEY, "0GB");
+ OmRatisSnapshotProvider provider =
+ new OmRatisSnapshotProvider(conf, snapshotDir, new HashMap<>());
+ assertDoesNotThrow(() -> provider.ensureBootstrapDiskSpace());
+ }
+
+ @Test
+ public void testBootstrapDiskSpaceCheckFailsWhenBelowMinimum(@TempDir File
snapshotDir) {
+ OzoneConfiguration conf = new OzoneConfiguration();
+ conf.set(OMConfigKeys.OZONE_OM_BOOTSTRAP_MIN_SPACE_KEY, "1024EB");
+ OmRatisSnapshotProvider provider =
+ new OmRatisSnapshotProvider(conf, snapshotDir, new HashMap<>());
+ IOException ex =
+ assertThrows(IOException.class, provider::ensureBootstrapDiskSpace);
+ assertEquals(true,
+
ex.getMessage().contains(OMConfigKeys.OZONE_OM_BOOTSTRAP_MIN_SPACE_KEY));
Review Comment:
Updated remaining assertions to use assertThat(...).contains(...) /
assertThat(...).isTrue() per HDDS-9951.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
