[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #986: HDDS-3476. Use persisted transaction info during OM startup in OM StateMachine.

2020-06-11 Thread GitBox 


hanishakoneru commented on a change in pull request #986:
URL: https://github.com/apache/hadoop-ozone/pull/986#discussion_r438922144



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java
##
@@ -338,20 +352,19 @@ public void unpause(long newLastAppliedSnaphsotIndex,
   }
 
   /**
-   * Take OM Ratis snapshot. Write the snapshot index to file. Snapshot index
-   * is the log index corresponding to the last applied transaction on the OM
-   * State Machine.
+   * Take OM Ratis snapshot is a dummy operation as when double buffer
+   * flushes the lastAppliedIndex is flushed to DB and that is used as
+   * snapshot index.
*
* @return the last applied index on the state machine which has been
* stored in the snapshot file.
*/
   @Override
   public long takeSnapshot() throws IOException {
-LOG.info("Saving Ratis snapshot on the OM.");
-if (ozoneManager != null) {
-  return ozoneManager.saveRatisSnapshot().getIndex();
-}
-return 0;
+LOG.info("Current Snapshot Index {}", getLastAppliedTermIndex());
+long lastAppliedIndex = getLastAppliedTermIndex().getIndex();
+ozoneManager.getMetadataManager().getStore().flush();

Review comment:
   The lastAppliedIndex returned to Ratis and updated in snapshotInfo could 
be different. We should call getLastAppliedTermIndex() only once and set the 
same value in snapshotInfo which is being returned to ratis.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #986: HDDS-3476. Use persisted transaction info during OM startup in OM StateMachine.

2020-06-10 Thread GitBox 


hanishakoneru commented on a change in pull request #986:
URL: https://github.com/apache/hadoop-ozone/pull/986#discussion_r438432283



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java
##
@@ -338,20 +352,19 @@ public void unpause(long newLastAppliedSnaphsotIndex,
   }
 
   /**
-   * Take OM Ratis snapshot. Write the snapshot index to file. Snapshot index
-   * is the log index corresponding to the last applied transaction on the OM
-   * State Machine.
+   * Take OM Ratis snapshot is a dummy operation as when double buffer
+   * flushes the lastAppliedIndex is flushed to DB and that is used as
+   * snapshot index.
*
* @return the last applied index on the state machine which has been
* stored in the snapshot file.
*/
   @Override
   public long takeSnapshot() throws IOException {
-LOG.info("Saving Ratis snapshot on the OM.");
-if (ozoneManager != null) {
-  return ozoneManager.saveRatisSnapshot().getIndex();
-}
-return 0;
+LOG.info("Current Snapshot Index {}", getLastAppliedTermIndex());
+long lastAppliedIndex = getLastAppliedTermIndex().getIndex();
+ozoneManager.getMetadataManager().getStore().flush();
+return lastAppliedIndex;

Review comment:
   So this needs to be fixed then. Or could lead to data loss.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #986: HDDS-3476. Use persisted transaction info during OM startup in OM StateMachine.

2020-06-10 Thread GitBox 


hanishakoneru commented on a change in pull request #986:
URL: https://github.com/apache/hadoop-ozone/pull/986#discussion_r438353082



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -259,16 +261,25 @@ public void start(OzoneConfiguration configuration) 
throws IOException {
 rocksDBConfiguration.setSyncOption(true);
   }
 
-  DBStoreBuilder dbStoreBuilder = DBStoreBuilder.newBuilder(configuration,
-  rocksDBConfiguration).setName(OM_DB_NAME)
-  .setPath(Paths.get(metaDir.getPath()));
+  this.store = loadDB(configuration, metaDir);
 
-  this.store = addOMTablesAndCodecs(dbStoreBuilder).build();
+  // This value will be used internally, not to be exposed to end users.

Review comment:
   We can remove this comment now.

##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3033,32 +3024,47 @@ public TermIndex installSnapshot(String leaderId) {
 DBCheckpoint omDBcheckpoint = getDBCheckpointFromLeader(leaderId);
 Path newDBlocation = omDBcheckpoint.getCheckpointLocation();
 
-// Check if current ratis log index is smaller than the downloaded
-// snapshot index. If yes, proceed by stopping the ratis server so that
-// the OM state can be re-initialized. If no, then do not proceed with
-// installSnapshot.
+LOG.info("Downloaded checkpoint from Leader {}, in to the location {}",
+leaderId, newDBlocation);
+
 long lastAppliedIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
-long checkpointSnapshotIndex = omDBcheckpoint.getRatisSnapshotIndex();
-long checkpointSnapshotTermIndex =
-omDBcheckpoint.getRatisSnapshotTerm();
-if (checkpointSnapshotIndex <= lastAppliedIndex) {
-  LOG.error("Failed to install checkpoint from OM leader: {}. The last " +
-  "applied index: {} is greater than or equal to the checkpoint's"
-  + " " +
-  "snapshot index: {}. Deleting the downloaded checkpoint {}",
-  leaderId,
-  lastAppliedIndex, checkpointSnapshotIndex,
+
+// Check if current ratis log index is smaller than the downloaded
+// checkpoint transaction index. If yes, proceed by stopping the ratis
+// server so that the OM state can be re-initialized. If no, then do not
+// proceed with installSnapshot.
+
+OMTransactionInfo omTransactionInfo = null;
+
+Path dbDir = newDBlocation.getParent();
+if (dbDir == null) {
+  LOG.error("Incorrect DB location path {} received from checkpoint.",
   newDBlocation);
-  try {
-FileUtils.deleteFully(newDBlocation);
-  } catch (IOException e) {
-LOG.error("Failed to fully delete the downloaded DB checkpoint {} " +
-"from OM leader {}.", newDBlocation,
-leaderId, e);
-  }
   return null;
 }
 
+try {
+  omTransactionInfo =
+  OzoneManagerRatisUtils.getTransactionInfoFromDownloadedSnapshot(
+  configuration, dbDir);
+} catch (Exception ex) {
+  LOG.error("Failed during opening downloaded snapshot from " +
+  "{} to obtain transaction index", newDBlocation, ex);
+  return null;
+}
+
+boolean canProceed =
+OzoneManagerRatisUtils.verifyTransactionInfo(omTransactionInfo,
+lastAppliedIndex, leaderId, newDBlocation);
+

Review comment:
   The lastAppliedIndex could have been updated between its assignment and 
the canProceed check. This check should be synchronous. Or at least the 
assignment should happen after reading the transactionInfo from DB.

##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3168,8 +3172,8 @@ File replaceOMDBWithCheckpoint(long lastAppliedIndex, 
Path checkpointPath)
* All the classes which use/ store MetadataManager should also be updated
* with the new MetadataManager instance.
*/
-  void reloadOMState(long newSnapshotIndex,
-  long newSnapShotTermIndex) throws IOException {
+  void reloadOMState(long newSnapshotIndex, long newSnapShotTermIndex)
+  throws IOException {

Review comment:
   NIT: SnapShot -> Snapshot

##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OzoneManagerSnapshotProvider.java
##
@@ -112,16 +112,16 @@ public OzoneManagerSnapshotProvider(ConfigurationSource 
conf,
*/
   public DBCheckpoint getOzoneManagerDBSnapshot(String leaderOMNodeID)
   throws IOException {
-String snapshotFileName = OM_SNAPSHOT_DB + "_" + 
System.currentTimeMillis();
-File targetFile = new File(omSnapshotDir, snapshotFileName + ".tar.gz");
+String snapshotTime = Long.toString(System.currentTimeMillis());
+String snapshotFileName = Paths.get(omSnapshotDir.getAbsolutePath(),
+snapshotTime,

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #986: HDDS-3476. Use persisted transaction info during OM startup in OM StateMachine.

2020-06-09 Thread GitBox 


hanishakoneru commented on a change in pull request #986:
URL: https://github.com/apache/hadoop-ozone/pull/986#discussion_r437643014



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3027,30 +3017,59 @@ public TermIndex installSnapshot(String leaderId) {
 DBCheckpoint omDBcheckpoint = getDBCheckpointFromLeader(leaderId);
 Path newDBlocation = omDBcheckpoint.getCheckpointLocation();
 
-// Check if current ratis log index is smaller than the downloaded
-// snapshot index. If yes, proceed by stopping the ratis server so that
-// the OM state can be re-initialized. If no, then do not proceed with
-// installSnapshot.
 long lastAppliedIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
-long checkpointSnapshotIndex = omDBcheckpoint.getRatisSnapshotIndex();
-long checkpointSnapshotTermIndex =
-omDBcheckpoint.getRatisSnapshotTerm();
-if (checkpointSnapshotIndex <= lastAppliedIndex) {
-  LOG.error("Failed to install checkpoint from OM leader: {}. The last " +
-  "applied index: {} is greater than or equal to the checkpoint's " +
-  "snapshot index: {}. Deleting the downloaded checkpoint {}", 
leaderId,
-  lastAppliedIndex, checkpointSnapshotIndex,
-  newDBlocation);
-  try {
-FileUtils.deleteFully(newDBlocation);
-  } catch (IOException e) {
-LOG.error("Failed to fully delete the downloaded DB checkpoint {} " +
-"from OM leader {}.", newDBlocation,
-leaderId, e);
+
+// Check if current ratis log index is smaller than the downloaded
+// checkpoint transaction index. If yes, proceed by stopping the ratis
+// server so that the OM state can be re-initialized. If no, then do not
+// proceed with installSnapshot.
+
+OMTransactionInfo omTransactionInfo = null;
+try {
+  // Set new DB location as DB path
+  OzoneConfiguration tempConfig = getConfiguration();
+
+  Path dbDir = newDBlocation.getParent();
+  if (dbDir != null) {
+tempConfig.set(OZONE_OM_DB_DIRS, dbDir.toString());
+  } else {
+LOG.error("Incorrect DB location path {} received from checkpoint.",
+newDBlocation);
+return null;
+  }
+
+  OMMetadataManager tempMetadataMgr =
+  new OmMetadataManagerImpl(configuration);

Review comment:
   Passing default config object should be trivial. Though this approach 
might make it simpler in terms of lines of code, it is not cleaner. I still 
don't see why OmMetadataMngr should be initialized just to read one value from 
the DB.
   Even better would be to pass this value from the leader itself. But not sure 
how complicated that can get. Might have to pause Leader disk sync while 
sending the checkpoint.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #986: HDDS-3476. Use persisted transaction info during OM startup in OM StateMachine.

2020-06-09 Thread GitBox 


hanishakoneru commented on a change in pull request #986:
URL: https://github.com/apache/hadoop-ozone/pull/986#discussion_r437073503



##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3027,30 +3017,59 @@ public TermIndex installSnapshot(String leaderId) {
 DBCheckpoint omDBcheckpoint = getDBCheckpointFromLeader(leaderId);
 Path newDBlocation = omDBcheckpoint.getCheckpointLocation();
 
-// Check if current ratis log index is smaller than the downloaded
-// snapshot index. If yes, proceed by stopping the ratis server so that
-// the OM state can be re-initialized. If no, then do not proceed with
-// installSnapshot.
 long lastAppliedIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
-long checkpointSnapshotIndex = omDBcheckpoint.getRatisSnapshotIndex();
-long checkpointSnapshotTermIndex =
-omDBcheckpoint.getRatisSnapshotTerm();
-if (checkpointSnapshotIndex <= lastAppliedIndex) {
-  LOG.error("Failed to install checkpoint from OM leader: {}. The last " +
-  "applied index: {} is greater than or equal to the checkpoint's " +
-  "snapshot index: {}. Deleting the downloaded checkpoint {}", 
leaderId,
-  lastAppliedIndex, checkpointSnapshotIndex,
-  newDBlocation);
-  try {
-FileUtils.deleteFully(newDBlocation);
-  } catch (IOException e) {
-LOG.error("Failed to fully delete the downloaded DB checkpoint {} " +
-"from OM leader {}.", newDBlocation,
-leaderId, e);
+
+// Check if current ratis log index is smaller than the downloaded
+// checkpoint transaction index. If yes, proceed by stopping the ratis
+// server so that the OM state can be re-initialized. If no, then do not
+// proceed with installSnapshot.
+
+OMTransactionInfo omTransactionInfo = null;
+try {
+  // Set new DB location as DB path
+  OzoneConfiguration tempConfig = getConfiguration();
+
+  Path dbDir = newDBlocation.getParent();
+  if (dbDir != null) {
+tempConfig.set(OZONE_OM_DB_DIRS, dbDir.toString());
+  } else {
+LOG.error("Incorrect DB location path {} received from checkpoint.",
+newDBlocation);
+return null;
+  }
+
+  OMMetadataManager tempMetadataMgr =
+  new OmMetadataManagerImpl(configuration);

Review comment:
   This still seems very hacky. My suggestion is to move the logic inside 
OMMetadataManagerImpl#start() to a new method loadDB(File dbPath) which returns 
DBStore object. 
   
   
   ```
 public void start(OzoneConfiguration configuration) throws IOException {
   // We need to create the DB here, as when during restart, stop closes the
   // db, so we need to create the store object and initialize DB.
   if (store == null) {
   File metaDir = OMStorage.getOmDbDir(configuration);
   store = loadDB(metaDir);
   initializeOmTables();
   }
 }
   ```
   
   And after downloading the checkpoint from Leader, we would have to just call 
thie loadDB method with the location of the downloaded checkpoint. This way we 
can also keep the format of the downloaded checkpoint as before.
   
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #986: HDDS-3476. Use persisted transaction info during OM startup in OM StateMachine.

2020-06-01 Thread GitBox 


hanishakoneru commented on a change in pull request #986:
URL: https://github.com/apache/hadoop-ozone/pull/986#discussion_r433429260



##
File path: 
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
##
@@ -133,13 +133,6 @@ private OMConfigKeys() {
   "ozone.om.ratis.log.purge.gap";
   public static final int OZONE_OM_RATIS_LOG_PURGE_GAP_DEFAULT = 100;
 
-  // OM Snapshot configurations
-  public static final String OZONE_OM_RATIS_SNAPSHOT_AUTO_TRIGGER_THRESHOLD_KEY
-  = "ozone.om.ratis.snapshot.auto.trigger.threshold";

Review comment:
   We need this config to be able to control the snapshot trigger from 
Ratis side for testing purposes. For some tests (TestOzoneManagerHA, OM HA 
robot tests) we need to set a lower value for auto trigger threshold so that 
logs can be purged which in turn can instantiate install snapshots. 
   We can keep it as an internal config and not expose it to users.

##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3027,30 +3017,59 @@ public TermIndex installSnapshot(String leaderId) {
 DBCheckpoint omDBcheckpoint = getDBCheckpointFromLeader(leaderId);
 Path newDBlocation = omDBcheckpoint.getCheckpointLocation();
 
-// Check if current ratis log index is smaller than the downloaded
-// snapshot index. If yes, proceed by stopping the ratis server so that
-// the OM state can be re-initialized. If no, then do not proceed with
-// installSnapshot.
 long lastAppliedIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
-long checkpointSnapshotIndex = omDBcheckpoint.getRatisSnapshotIndex();
-long checkpointSnapshotTermIndex =
-omDBcheckpoint.getRatisSnapshotTerm();
-if (checkpointSnapshotIndex <= lastAppliedIndex) {
-  LOG.error("Failed to install checkpoint from OM leader: {}. The last " +
-  "applied index: {} is greater than or equal to the checkpoint's " +
-  "snapshot index: {}. Deleting the downloaded checkpoint {}", 
leaderId,
-  lastAppliedIndex, checkpointSnapshotIndex,
-  newDBlocation);
-  try {
-FileUtils.deleteFully(newDBlocation);
-  } catch (IOException e) {
-LOG.error("Failed to fully delete the downloaded DB checkpoint {} " +
-"from OM leader {}.", newDBlocation,
-leaderId, e);
+
+// Check if current ratis log index is smaller than the downloaded
+// checkpoint transaction index. If yes, proceed by stopping the ratis
+// server so that the OM state can be re-initialized. If no, then do not
+// proceed with installSnapshot.
+
+OMTransactionInfo omTransactionInfo = null;
+try {
+  // Set new DB location as DB path
+  OzoneConfiguration tempConfig = getConfiguration();
+
+  Path dbDir = newDBlocation.getParent();
+  if (dbDir != null) {
+tempConfig.set(OZONE_OM_DB_DIRS, dbDir.toString());
+  } else {
+LOG.error("Incorrect DB location path {} received from checkpoint.",
+newDBlocation);
+return null;
+  }
+
+  OMMetadataManager tempMetadataMgr =
+  new OmMetadataManagerImpl(configuration);

Review comment:
   Instead of instantiating a new MetadataManagerImpl, can we not directly 
load the RocksDB and read the transactionInfo?

##
File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3027,30 +3017,59 @@ public TermIndex installSnapshot(String leaderId) {
 DBCheckpoint omDBcheckpoint = getDBCheckpointFromLeader(leaderId);
 Path newDBlocation = omDBcheckpoint.getCheckpointLocation();
 
-// Check if current ratis log index is smaller than the downloaded
-// snapshot index. If yes, proceed by stopping the ratis server so that
-// the OM state can be re-initialized. If no, then do not proceed with
-// installSnapshot.
 long lastAppliedIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
-long checkpointSnapshotIndex = omDBcheckpoint.getRatisSnapshotIndex();
-long checkpointSnapshotTermIndex =
-omDBcheckpoint.getRatisSnapshotTerm();
-if (checkpointSnapshotIndex <= lastAppliedIndex) {
-  LOG.error("Failed to install checkpoint from OM leader: {}. The last " +
-  "applied index: {} is greater than or equal to the checkpoint's " +
-  "snapshot index: {}. Deleting the downloaded checkpoint {}", 
leaderId,
-  lastAppliedIndex, checkpointSnapshotIndex,
-  newDBlocation);
-  try {
-FileUtils.deleteFully(newDBlocation);
-  } catch (IOException e) {
-LOG.error("Failed to fully delete the downloaded DB checkpoint {} " +
-"from OM leader {}.", newDBlocation,
-leaderId, e);
+
+// Check if current ratis log index is smaller than the downloaded
+// checkpoint transaction index. If yes, proceed by stopping the