sodonnel commented on a change in pull request #719: HDDS-3270. Allow safemode
listeners to be notified when some precheck rules pass
URL: https://github.com/apache/hadoop-ozone/pull/719#discussion_r400514057
##########
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##########
@@ -641,13 +663,26 @@ public boolean getSafeModeStatus() {
}
@Override
- public void handleSafeModeTransition(
+ public synchronized void handleSafeModeTransition(
SCMSafeModeManager.SafeModeStatus status) {
- this.isInSafeMode.set(status.getSafeModeStatus());
- if (!status.getSafeModeStatus()) {
- // TODO: #CLUTIL if we reenter safe mode the fixed interval pipeline
- // creation job needs to stop
+ // TODO: #CLUTIL - handle safemode getting re-enabled
+ boolean currentAllowPipelines =
+ allowPipelineCreation.getAndSet(status.isPreCheckComplete());
+ boolean currentlyInSafeMode =
+ isInSafeMode.getAndSet(status.isInSafeMode());
+ boolean triggerPipelines = false;
+
+ // Trigger pipeline creation only if the preCheck status has changed to
+ // complete.
+ if (allowPipelineCreation.get() && !currentAllowPipelines) {
+ triggerPipelines = true;
+ }
+ // Start the pipeline creation thread only when safemode switches off
+ if (!getSafeModeStatus() && currentlyInSafeMode) {
startPipelineCreator();
+ triggerPipelines = true;
+ }
+ if (triggerPipelines) {
Review comment:
In the logic prior to this patch, `BackgroundPipelineCreator` only gets
started when safemode exits. However, the creator does not need to be started
for pipelines to get created. Infact, in order to exit safemode, there must be
a replication factor 3 pipeline created, and that must happen without the
creator running.
There are actually two ways a pipeline can be created:
1. When the backgroundPipelineCreator is running, it starts every 2 minutes
and tries to create pipelines.
2. When a node registers with SCM, it triggers an event and calls
`backgroundPipelineCreator.triggerPipelineCreation();` which tries to create
pipelines immediately - even if the BackgroundPipelineCreator has already been
started.
Therefore in safemode, before this patch, pipelines are initially created by
(2) above. However now we want to stop that happening. Therefore whenever the
precheck completes (enough nodes registered) we trigger the pipeline creation,
but hold off on starting the background creator thread. Then it is started when
safemode exits.
Now that you have highlighted this, it is possibly valid to just start the
BackgroundPipeLineCreator in both cases. However there is one further detail -
in the existing implementation, there is a delay for safemode exiting. Once the
rules have passed some services are not started for a delay of 5 minutes and
the BackgroundPipelineCreator thread is one of them. I am not sure on the
history of that so it probably makes sense to keep that logic for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
