avijayanhwx commented on a change in pull request #1998:
URL: https://github.com/apache/ozone/pull/1998#discussion_r606364359
##########
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/upgrade/BasicUpgradeFinalizer.java
##########
@@ -436,4 +456,15 @@ private void logAndThrow(Exception e, String msg,
ResultCodes resultCode)
protected void updateLayoutVersionInDB(V vm, T comp) throws IOException {
throw new UnsupportedOperationException();
}
+
+ protected void postFinalizeUpgrade() throws IOException {
+ }
+
+ protected void finalizeVersionManager(Storage storageConfig)
+ throws UpgradeException {
+ }
+
+ protected boolean preFinalizeUpgrade() throws IOException {
Review comment:
It seems like there is some overlap of logic between
**preFinalizeUpgrade()** in various components, and
org.apache.hadoop.ozone.upgrade.BasicUpgradeFinalizer#preFinalize. The
preFinalize() step can be called either by the finalizer or the
finalizationExecutor (based on whether we need that injection point), but in
that implementation, we should be able to check the need to finalize as well as
do pre-steps. If needed, we can split these 2 steps into needsFinalization()
and preFinalize() steps, and call it from one place, the finalizationExecutor.
##########
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/upgrade/UpgradeFinalizationExecutor.java
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.upgrade;
+
+import static
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+import org.apache.hadoop.ozone.common.Storage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * UpgradeFinalizationExecutor for driving the main part of finalization.
+ * Unit/Integration tests can override this to provide error injected version
+ * of this class.
+ */
+
+@SuppressWarnings("checkstyle:VisibilityModifier")
+public class UpgradeFinalizationExecutor {
+ static final Logger LOG =
+ LoggerFactory.getLogger(UpgradeFinalizationExecutor.class);
+
+ public UpgradeFinalizationExecutor() {
+ }
+
+ public Void execute(Storage storageConfig,
+ BasicUpgradeFinalizer basicUpgradeFinalizer)
+ throws Exception {
+ try {
+ basicUpgradeFinalizer.emitStartingMsg();
+ basicUpgradeFinalizer.getVersionManager()
+ .setUpgradeState(FINALIZATION_IN_PROGRESS);
+
+ /*
+ * Before we can call finalize the feature, we need to make sure that
+ * all existing pipelines are closed and pipeline Manger would freeze
+ * all new pipeline creation.
+ */
+ if(!basicUpgradeFinalizer.preFinalizeUpgrade()) {
+ return null;
+ }
+
+ basicUpgradeFinalizer.finalizeVersionManager(storageConfig);
+
+ basicUpgradeFinalizer.getVersionManager().completeFinalization();
Review comment:
Is it possible to absorb this into the last step
(finalizeVersionManager)? It seems out of place in the pre => finalize => post
flow.
##########
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/upgrade/InjectedUpgradeFinalizationExecutor.java
##########
@@ -0,0 +1,137 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.upgrade;
+
+import static
org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor.UpgradeTestInjectionPoints.AFTER_COMPLETE_FINALIZATION;
+import static
org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor.UpgradeTestInjectionPoints.AFTER_POST_FINALIZE_UPGRADE;
+import static
org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor.UpgradeTestInjectionPoints.AFTER_PRE_FINALIZE_UPGRADE;
+import static
org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor.UpgradeTestInjectionPoints.BEFORE_COMPLETE_FINALIZATION;
+import static
org.apache.hadoop.ozone.upgrade.InjectedUpgradeFinalizationExecutor.UpgradeTestInjectionPoints.BEFORE_PRE_FINALIZE_UPGRADE;
+import static
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.ozone.common.Storage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Failure injected extension of UpgradeFinalizationExecutor that can be used
by
+ * Unit/Integration Tests.
+ */
+@SuppressWarnings("checkstyle:VisibilityModifier")
+public class InjectedUpgradeFinalizationExecutor extends
Review comment:
Can we move this to the corresponding test package?
##########
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/upgrade/BasicUpgradeFinalizer.java
##########
@@ -436,4 +456,15 @@ private void logAndThrow(Exception e, String msg,
ResultCodes resultCode)
protected void updateLayoutVersionInDB(V vm, T comp) throws IOException {
throw new UnsupportedOperationException();
}
+
+ protected void postFinalizeUpgrade() throws IOException {
+ }
+
+ protected void finalizeVersionManager(Storage storageConfig)
Review comment:
nit. Can the method name be just finalizeUpgrade (to go with pre and
post)?
##########
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/upgrade/BasicUpgradeFinalizer.java
##########
@@ -57,18 +57,39 @@
protected V versionManager;
protected String clientID;
protected T component;
+ protected UpgradeFinalizationExecutor finalizationExecutor;
private Queue<String> msgs = new ConcurrentLinkedQueue<>();
protected boolean isDone = false;
public BasicUpgradeFinalizer(V versionManager) {
this.versionManager = versionManager;
+ this.finalizationExecutor =
+ new UpgradeFinalizationExecutor();
+ }
+
+ @Override
+ public void setFinalizationExecutor(UpgradeFinalizationExecutor executor) {
+ finalizationExecutor = executor;
+ }
+
+ @Override
+ public UpgradeFinalizationExecutor getFinalizationExecutor() {
+ return finalizationExecutor;
}
public boolean isFinalizationDone() {
return isDone;
}
+ public void markFinalizationDone() {
+ isDone = true;
Review comment:
Can we not have **state** here? Why cannot we refer to the version
manager to figure out that finalization is done or not?
The way I see it, the version manager maintains state. The upgrade finalizer
provides component specific upgrade finalization implementations (without
state), and the executor uses both of them to orchestrate the upgrade.
##########
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/upgrade/UpgradeFinalizationExecutor.java
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.upgrade;
+
+import static
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_IN_PROGRESS;
+import static
org.apache.hadoop.ozone.upgrade.UpgradeFinalizer.Status.FINALIZATION_REQUIRED;
+
+import org.apache.hadoop.ozone.common.Storage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * UpgradeFinalizationExecutor for driving the main part of finalization.
+ * Unit/Integration tests can override this to provide error injected version
+ * of this class.
+ */
+
+@SuppressWarnings("checkstyle:VisibilityModifier")
+public class UpgradeFinalizationExecutor {
Review comment:
nit. Maybe we can call it DefaultUpgradeFinalizationExecutor ?
##########
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/upgrade/BasicUpgradeFinalizer.java
##########
@@ -436,4 +456,15 @@ private void logAndThrow(Exception e, String msg,
ResultCodes resultCode)
protected void updateLayoutVersionInDB(V vm, T comp) throws IOException {
throw new UnsupportedOperationException();
}
+
+ protected void postFinalizeUpgrade() throws IOException {
+ }
+
+ protected void finalizeVersionManager(Storage storageConfig)
Review comment:
Given that we are refactoring the abstractions here, I think we can also
make org.apache.hadoop.ozone.upgrade.BasicUpgradeFinalizer#finalize abstract
and mandate subclasses to implement it.
##########
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/upgrade/UpgradeFinalizer.java
##########
@@ -187,4 +186,15 @@ StatusAndMessages reportStatus(String upgradeClientId,
boolean takeover)
*/
void runPrefinalizeStateActions(Storage storage, T service)
throws IOException;
+
+ /**
+ * Sets the Finalization Executor driver.
+ * @param executor FinalizationExecutor.
+ */
+ void setFinalizationExecutor(UpgradeFinalizationExecutor executor);
Review comment:
Can we create overriden constructors to the finalizers (DN, SCM) that
take in a custom executor? We can add one with a new executor parameter. If not
passed in, the default executor will be used. In that case, we can remove the
setter in the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
