hanishakoneru commented on a change in pull request #2491:
URL: https://github.com/apache/ozone/pull/2491#discussion_r745167605
##########
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -1453,7 +1559,13 @@ public void updatePeerList(List<String> omNodeIds) {
// Check if the OM NodeID is already present in the peer list or its
// the local NodeID.
if (!peerNodesMap.containsKey(omNodeId) && !isCurrentNode(omNodeId)) {
- addOMNodeToPeers(omNodeId);
+ try {
+ addOMNodeToPeers(omNodeId);
+ } catch (IOException e) {
+ LOG.error("Fatal Error: Shutting down the system as otherwise it " +
Review comment:
This function is called from OM SM. Any failure here could be fatal as
OM states would diverge.
For example, let's say OM1 and OM2 successfully update their configs with
newly bootstrapped nodes OM4 and OM5. OM3 fails to do so. Let's say later OM1
and OM2 shutdown with OM4 as the new leader. OM3 would not be able to download
checkpoint from OM4 as it is not aware of OM4 addresses.
##########
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerBootstrap.java
##########
@@ -205,12 +191,146 @@ public void testLeaderChangeToNewOM() throws Exception {
"other OMs are down", newOMNodeIds.contains(omLeader.getOMNodeId()));
// Perform some read and write operations with new OM leader
- objectStore = OzoneClientFactory.getRpcClient(OM_SERVICE_ID, conf)
- .getObjectStore();
+ objectStore = OzoneClientFactory.getRpcClient(OM_SERVICE_ID,
+ cluster.getConf()).getObjectStore();
+
OzoneVolume volume = objectStore.getVolume(VOLUME_NAME);
OzoneBucket bucket = volume.getBucket(BUCKET_NAME);
String key = createKey(bucket);
Assert.assertNotNull(bucket.getKey(key));
}
+
+ /**
+ * Tests the following scenarios:
+ * 1. Bootstrap without updating config on any existing OM -> fail
+ * 2. Force bootstrap without upating config on any OM -> fail
+ */
+ @Test
+ public void testBootstrapWithoutConfigUpdate() throws Exception {
+ // Setup 1 node cluster
+ setupCluster(1);
+ cluster.setupExitManagerForTesting();
+ OzoneManager existingOM = cluster.getOzoneManager(0);
+ String existingOMNodeId = existingOM.getOMNodeId();
+
+ GenericTestUtils.LogCapturer omLog =
+ GenericTestUtils.LogCapturer.captureLogs(OzoneManager.LOG);
+ GenericTestUtils.LogCapturer miniOzoneClusterLog =
+ GenericTestUtils.LogCapturer.captureLogs(MiniOzoneHAClusterImpl.LOG);
+
+
/***************************************************************************
+ * 1. Bootstrap without updating config on any existing OM -> fail
+
**************************************************************************/
+
+ // Bootstrap a new node without updating the configs on existing OMs.
+ // This should result in the bootstrap failing.
+ String newNodeId = "omNode-bootstrap-1";
+ try {
+ cluster.bootstrapOzoneManager(newNodeId, false, false);
+ Assert.fail("Bootstrap should have failed as configs are not updated on"
+
+ " all OMs.");
+ } catch (Exception e) {
+ Assert.assertEquals(OmUtils.getOMAddressListPrintString(
+ Lists.newArrayList(existingOM.getNodeDetails())) + " do not have or"
+
+ " have incorrect information of the bootstrapping OM. Update their "
+
+ "ozone-site.xml before proceeding.", e.getMessage());
+ Assert.assertTrue(omLog.getOutput().contains("Remote OM config check " +
+ "failed on OM " + existingOMNodeId));
+ Assert.assertTrue(miniOzoneClusterLog.getOutput().contains(newNodeId +
+ " - System Exit"));
+ }
+
+
/***************************************************************************
+ * 2. Force bootstrap without upating config on any OM -> fail
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
