smengcl commented on code in PR #10185:
URL: https://github.com/apache/ozone/pull/10185#discussion_r3555940919
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -4110,6 +4113,16 @@ public synchronized TermIndex
installSnapshotFromLeader(String leaderId) throws
omDBCheckpoint = omRatisSnapshotProvider.
downloadDBSnapshotFromLeader(leaderId);
} catch (IOException ex) {
+ if (OmRatisSnapshotProvider.isDiskFullOrQuotaIOException(ex)) {
+ LOG.error(
+ "Failed to download snapshot from leader {}: local disk appears
full or over quota "
+ + "on the OM ratis snapshot volume (see previous ERROR for
path/usable space). "
+ + "Free disk or adjust {}, {}, or {} before bootstrap can
succeed.",
+ leaderId,
+ OZONE_OM_BOOTSTRAP_MIN_SPACE_KEY,
+ OZONE_OM_BOOTSTRAP_CHECKPOINT_HEADROOM_RATIO_KEY,
+ OZONE_OM_CHECKPOINT_ESTIMATED_SST_BYTES_HEADER);
+ }
Review Comment:
Each disk-full failure would be logged 3 times:
1. this block
2. `logDiskFullOrQuotaDuringDownload` in the provider
3. `LOG.error("Failed to download snapshot from Leader", ex)` below
Ideally we should log only once for each failure. CMIIW
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis_snapshot/OmRatisSnapshotProvider.java:
##########
@@ -86,6 +96,88 @@ public class OmRatisSnapshotProvider extends
RDBSnapshotProvider {
private final boolean spnegoEnabled;
private final URLConnectionFactory connectionFactory;
private final boolean useV2CheckpointApi;
+ /** Minimum usable bytes on snapshot volume before download; 0 = disabled. */
+ private final long bootstrapMinSpaceBytes;
+ /** Applied to leader-reported SST byte estimate to reserve tar/unpack
headroom. */
+ private final double bootstrapCheckpointHeadroomRatio;
+
+ private static final class BootstrapSpaceRequirement {
+ private final long requiredBytes;
+ private final boolean usedLeaderEstimateHeader;
+
+ private BootstrapSpaceRequirement(long requiredBytes, boolean
usedLeaderEstimateHeader) {
+ this.requiredBytes = requiredBytes;
+ this.usedLeaderEstimateHeader = usedLeaderEstimateHeader;
+ }
+ }
+
+ /**
+ * Whether this {@link IOException} (or its causes) typically means the
+ * local filesystem ran out of space or hit a quota while writing.
+ */
+ public static boolean isDiskFullOrQuotaIOException(IOException ioe) {
+ for (Throwable t = ioe; t != null; t = t.getCause()) {
+ if (t instanceof FileSystemException) {
+ FileSystemException fse = (FileSystemException) t;
+ String reason = fse.getReason();
+ if (reason != null) {
+ String r = reason.toLowerCase(Locale.ROOT);
+ if (r.contains("no space") || r.contains("space left")
+ || r.contains("quota") || r.contains("enospc")) {
+ return true;
+ }
+ }
+ }
+ String msg = t.getMessage();
+ if (msg != null) {
+ String m = msg.toLowerCase(Locale.ROOT);
+ if (m.contains("no space left on device")
+ || m.contains("enospc")
+ || m.contains("disk quota exceeded")
+ || m.contains("quota exceeded")) {
+ return true;
Review Comment:
String-matching English substrings might miss localized error messages in
other languages?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
