devmadhuu commented on code in PR #11264:
URL: https://github.com/apache/ozone/pull/11264#discussion_r4053738493
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java:
##########
@@ -572,6 +576,62 @@ public void start(String clusterId) throws IOException {
}
}
+ /**
+ * Runs {@link #initializeContainerServices(String)}, optionally bounded by a
+ * watchdog timeout ({@code hdds.datanode.container.init.timeout}).
+ * <p>
+ * When the timeout is a positive duration, initialization runs on a separate
+ * thread so a stall - for example Ratis group recovery blocked on a failing
+ * volume inside {@code writeChannel.start()} - is turned into an
+ * {@link IOException} instead of an indefinite hang. The caller ({@code
+ * start}) then records the failure ({@code FAILED}) and the datanode shuts
+ * down cleanly. A non-positive timeout disables the watchdog and runs
+ * initialization inline, preserving the previous behavior.
+ */
+ private void initializeContainerServicesWithTimeout(String clusterId) throws
IOException {
+ Duration initTimeout =
+
config.getObject(DatanodeConfiguration.class).getContainerInitTimeout();
+ if (initTimeout == null || initTimeout.isZero() ||
initTimeout.isNegative()) {
+ initializeContainerServices(clusterId);
+ return;
+ }
+
+ ExecutorService initExecutor = Executors.newSingleThreadExecutor(
+ new ThreadFactoryBuilder().setDaemon(true)
+ .setNameFormat("OzoneContainerInit").build());
+ Future<?> initFuture = initExecutor.submit(() -> {
+ initializeContainerServices(clusterId);
+ return null;
+ });
+ try {
+ initFuture.get(initTimeout.toMillis(), TimeUnit.MILLISECONDS);
+ } catch (TimeoutException e) {
+ // Best-effort interrupt; a thread blocked on disk I/O may not respond,
+ // but the datanode will shut down once the caller marks startup FAILED.
+ initFuture.cancel(true);
+ throw new IOException("OzoneContainer initialization did not complete
within " + initTimeout
+ + ". Failing datanode startup; a stalled Ratis group recovery or
volume I/O is the likely cause.", e);
Review Comment:
Good catch, thanks — you're right, and I confirmed it.
`cancel(true)/shutdownNow()` only interrupt the worker, and the interrupt is
ignored by the stalled Ratis `RaftServerProxy.startImpl()
CompletableFuture.join()`. I have handled in different way using separate watch
thread. Kindly have a re-look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
