[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
prashantpogde commented on a change in pull request #1392: URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r485313969 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/upgrade/HDDSLayoutVersionManager.java ## @@ -0,0 +1,92 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.upgrade; + + +import java.io.IOException; + +import org.apache.hadoop.ozone.common.Storage; +import org.apache.hadoop.hdds.upgrade.HDDSLayoutFeatureCatalog.HDDSLayoutFeature; +import org.apache.hadoop.ozone.upgrade.AbstractLayoutVersionManager; +import org.apache.hadoop.ozone.upgrade.LayoutVersionManager; + +import com.google.common.annotations.VisibleForTesting; + +/** + * Class to manage layout versions and features for Storage Container Manager + * and DataNodes. + */ +public final class HDDSLayoutVersionManager extends Review comment: I will add the test in another PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
prashantpogde commented on a change in pull request #1392: URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r485203276 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/upgrade/NewSCMFeatureUpgradeAction.java ## @@ -0,0 +1,34 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.scm.server.upgrade; + +import org.apache.hadoop.hdds.scm.server.StorageContainerManager; +import org.apache.hadoop.hdds.upgrade.HDDSUpgradeAction; + +/** + * Example SCM Action class to help with understanding. + */ +public class NewSCMFeatureUpgradeAction implements Review comment: Yes, left it as an example. Can we remove it after we are done with other upgrade changes too ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
prashantpogde commented on a change in pull request #1392: URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r485202562 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/upgrade/HDDSLayoutVersionManager.java ## @@ -0,0 +1,92 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.upgrade; + + +import java.io.IOException; + +import org.apache.hadoop.ozone.common.Storage; +import org.apache.hadoop.hdds.upgrade.HDDSLayoutFeatureCatalog.HDDSLayoutFeature; +import org.apache.hadoop.ozone.upgrade.AbstractLayoutVersionManager; +import org.apache.hadoop.ozone.upgrade.LayoutVersionManager; + +import com.google.common.annotations.VisibleForTesting; + +/** + * Class to manage layout versions and features for Storage Container Manager + * and DataNodes. + */ +public final class HDDSLayoutVersionManager extends Review comment: yup This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
prashantpogde commented on a change in pull request #1392:
URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r485202252
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/exceptions/SCMException.java
##
@@ -123,6 +123,7 @@ public ResultCodes getResult() {
FAILED_TO_INIT_CONTAINER_PLACEMENT_POLICY,
FAILED_TO_ALLOCATE_ENOUGH_BLOCKS,
INTERNAL_ERROR,
-FAILED_TO_INIT_PIPELINE_CHOOSE_POLICY
+FAILED_TO_INIT_PIPELINE_CHOOSE_POLICY,
+NOT_SUPPORTED_OPERATION,
Review comment:
yup
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
prashantpogde commented on a change in pull request #1392:
URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r484708634
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -2244,6 +2248,14 @@ public OmKeyInfo lookupKey(OmKeyArgs args) throws
IOException {
}
}
+
+ @Override
+ public void renameKeys(OmRenameKeys omRenameKeys)
Review comment:
NA, like above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
prashantpogde commented on a change in pull request #1392:
URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r484708258
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/storage/ContainerProtocolCalls.java
##
@@ -578,4 +581,35 @@ private static Text getService(DatanodeBlockID blockId) {
validators.add(validator);
return validators;
}
+
+ public static HashMap
+ getBlockFromAllNodes(
Review comment:
not sure, why these changes are showed up in this PR. These changes were
not made as part of this PR.
I have re-uploaded the changes and it should go away.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
prashantpogde commented on a change in pull request #1392:
URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r484708528
##
File path:
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ScmSubcommand.java
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm.cli;
+
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.concurrent.Callable;
+
+/**
+ * Base class for admin commands that connect via SCM client.
+ */
+public abstract class ScmSubcommand implements Callable {
+
+ @CommandLine.Mixin
+ private ScmOption scmOption;
+
+ protected abstract void execute(ScmClient client) throws IOException;
Review comment:
NA. Please see the comment above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
prashantpogde commented on a change in pull request #1392:
URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r484706946
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/upgrade/SCMLayoutFeatureCatalog.java
##
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.upgrade;
+
+import java.util.Optional;
+
+import org.apache.hadoop.ozone.upgrade.LayoutFeature;
+
+/**
+ * Catalog of SCM features.
+ */
+public class SCMLayoutFeatureCatalog {
+
+ /**
+ * List of SCM Features.
+ */
+ public enum SCMLayoutFeature implements LayoutFeature {
+INITIAL_VERSION(0, "Initial Layout Version"),
+CREATE_EC(1, ""),
+NEW_FEATURE(2, "new feature", new NewSCMFeatureUpgradeAction());
+
+
+private int layoutVersion;
+private String description;
+private Optional scmUpgradeAction = Optional.empty();
+
+SCMLayoutFeature(final int layoutVersion, String description) {
+ this.layoutVersion = layoutVersion;
+ this.description = description;
+}
+
+SCMLayoutFeature(final int layoutVersion, String description,
+SCMUpgradeAction upgradeAction) {
+ this.layoutVersion = layoutVersion;
+ this.description = description;
+ scmUpgradeAction = Optional.of(upgradeAction);
+}
+
+@Override
+public int layoutVersion() {
+ return layoutVersion;
+}
+
+@Override
+public String description() {
+ return description;
+}
+
+@Override
+public Optional onFinalizeAction() {
+ return scmUpgradeAction;
+}
+ }
+
+ /**
+ * This is an example of an "API" that uses a new Layout feature (EC) that is
+ * not yet supported by the current layout version. The following can be
+ * "guarded" by just adding the following annotation, thereby keeping the
+ * method logic and upgrade logic separate.
+ */
+ @SCMLayoutFeatureAPI(SCMLayoutFeature.CREATE_EC)
Review comment:
done
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/upgrade/SCMLayoutFeatureCatalog.java
##
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.upgrade;
+
+import java.util.Optional;
+
+import org.apache.hadoop.ozone.upgrade.LayoutFeature;
+
+/**
+ * Catalog of SCM features.
+ */
+public class SCMLayoutFeatureCatalog {
+
+ /**
+ * List of SCM Features.
+ */
+ public enum SCMLayoutFeature implements LayoutFeature {
+INITIAL_VERSION(0, "Initial Layout Version"),
+CREATE_EC(1, ""),
+NEW_FEATURE(2, "new feature", new NewSCMFeatureUpgradeAction());
+
+
+private int layoutVersion;
+private String description;
+private Optional scmUpgradeAction = Optional.empty();
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1392: HDDS-4173. Implement HDDS Version management using the LayoutVersion…
prashantpogde commented on a change in pull request #1392:
URL: https://github.com/apache/hadoop-ozone/pull/1392#discussion_r484706750
##
File path: hadoop-hdds/server-scm/pom.xml
##
@@ -29,6 +29,19 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd;>
jar
+
+
Review comment:
done
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/upgrade/SCMLayoutFeatureAspect.java
##
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.upgrade;
+
+import static
org.apache.hadoop.hdds.scm.exceptions.SCMException.ResultCodes.NOT_SUPPORTED_OPERATION;
+
+import org.apache.hadoop.hdds.scm.exceptions.SCMException;
+import org.apache.hadoop.ozone.upgrade.LayoutFeature;
+import org.apache.hadoop.ozone.upgrade.LayoutVersionManager;
+import org.aspectj.lang.JoinPoint;
+import org.aspectj.lang.annotation.Aspect;
+import org.aspectj.lang.annotation.Before;
+import org.aspectj.lang.reflect.MethodSignature;
+
+/**
+ * 'Aspect' for SCM Layout Feature API. All methods annotated with the
+ * specific annotation will have pre-processing done here to check layout
+ * version compatibility.
+ */
+@Aspect
+public class SCMLayoutFeatureAspect {
Review comment:
done
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/upgrade/SCMLayoutFeatureCatalog.java
##
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.upgrade;
+
+import java.util.Optional;
+
+import org.apache.hadoop.ozone.upgrade.LayoutFeature;
+
+/**
+ * Catalog of SCM features.
+ */
+public class SCMLayoutFeatureCatalog {
+
+ /**
+ * List of SCM Features.
+ */
+ public enum SCMLayoutFeature implements LayoutFeature {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
