avijayanhwx commented on a change in pull request #1259: HDDS-1105 : Add
mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r314932606
##########
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java
##########
@@ -187,5 +229,119 @@ protected DBCheckpoint getOzoneManagerDBSnapshot() {
}
return null;
}
+
+ /**
+ * Update Local OM DB with new OM DB snapshot.
+ * @throws IOException
+ */
+ @VisibleForTesting
+ void updateReconOmDBWithNewSnapshot() throws IOException {
+ // Obtain the current DB snapshot from OM and
+ // update the in house OM metadata managed DB instance.
+ DBCheckpoint dbSnapshot = getOzoneManagerDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ try {
+ omMetadataManager.updateOmDB(dbSnapshot.getCheckpointLocation()
+ .toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon OM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from OM.");
+ }
+ }
+
+ /**
+ * Get Delta updates from OM through RPC call and apply to local OM DB as
+ * well as accumulate in a buffer.
+ * @param fromSequenceNumber from sequence number to request from.
+ * @param omdbUpdatesHandler OM DB updates handler to buffer updates.
+ * @throws IOException when OM RPC request fails.
+ * @throws RocksDBException when writing to RocksDB fails.
+ */
+ @VisibleForTesting
+ void getAndApplyDeltaUpdatesFromOM(
+ long fromSequenceNumber, OMDBUpdatesHandler omdbUpdatesHandler)
+ throws IOException, RocksDBException {
+ DBUpdatesRequest dbUpdatesRequest = DBUpdatesRequest.newBuilder()
+ .setSequenceNumber(fromSequenceNumber).build();
+ DBUpdatesWrapper dbUpdates = ozoneManagerClient.getDBUpdates(
+ dbUpdatesRequest);
+ if (null != dbUpdates) {
+ RDBStore rocksDBStore = (RDBStore)omMetadataManager.getStore();
+ RocksDB rocksDB = rocksDBStore.getDb();
+ LOG.debug("Number of updates received from OM : " +
+ dbUpdates.getData().size());
+ for (byte[] data : dbUpdates.getData()) {
+ WriteBatch writeBatch = new WriteBatch(data);
+ writeBatch.iterate(omdbUpdatesHandler);
+ RDBBatchOperation rdbBatchOperation = new
RDBBatchOperation(writeBatch);
+ rdbBatchOperation.commit(rocksDB, new WriteOptions());
+ }
+ }
+ }
+
+ /**
+ * Based on current state of Recon's OM DB, we either get delta updates or
+ * full snapshot from Ozone Manager.
+ */
+ @VisibleForTesting
+ void syncDataFromOM() {
+ long currentSequenceNumber = getCurrentOMDBSequenceNumber();
+ boolean fullSnapshot = false;
+
+ if (currentSequenceNumber <= 0) {
+ fullSnapshot = true;
+ } else {
Review comment:
Good question @hanishakoneru . This is handled in OM side. Please check out
org.apache.hadoop.utils.db.RDBStore#getUpdatesSince.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
