[
https://issues.apache.org/jira/browse/HDDS-1105?focusedWorklogId=296708&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296708
]
ASF GitHub Bot logged work on HDDS-1105:
----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Aug/19 03:19
Start Date: 17/Aug/19 03:19
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1259: HDDS-1105
: Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
Manager
URL: https://github.com/apache/hadoop/pull/1259#discussion_r314932606
##########
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java
##########
@@ -187,5 +229,119 @@ protected DBCheckpoint getOzoneManagerDBSnapshot() {
}
return null;
}
+
+ /**
+ * Update Local OM DB with new OM DB snapshot.
+ * @throws IOException
+ */
+ @VisibleForTesting
+ void updateReconOmDBWithNewSnapshot() throws IOException {
+ // Obtain the current DB snapshot from OM and
+ // update the in house OM metadata managed DB instance.
+ DBCheckpoint dbSnapshot = getOzoneManagerDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ try {
+ omMetadataManager.updateOmDB(dbSnapshot.getCheckpointLocation()
+ .toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon OM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from OM.");
+ }
+ }
+
+ /**
+ * Get Delta updates from OM through RPC call and apply to local OM DB as
+ * well as accumulate in a buffer.
+ * @param fromSequenceNumber from sequence number to request from.
+ * @param omdbUpdatesHandler OM DB updates handler to buffer updates.
+ * @throws IOException when OM RPC request fails.
+ * @throws RocksDBException when writing to RocksDB fails.
+ */
+ @VisibleForTesting
+ void getAndApplyDeltaUpdatesFromOM(
+ long fromSequenceNumber, OMDBUpdatesHandler omdbUpdatesHandler)
+ throws IOException, RocksDBException {
+ DBUpdatesRequest dbUpdatesRequest = DBUpdatesRequest.newBuilder()
+ .setSequenceNumber(fromSequenceNumber).build();
+ DBUpdatesWrapper dbUpdates = ozoneManagerClient.getDBUpdates(
+ dbUpdatesRequest);
+ if (null != dbUpdates) {
+ RDBStore rocksDBStore = (RDBStore)omMetadataManager.getStore();
+ RocksDB rocksDB = rocksDBStore.getDb();
+ LOG.debug("Number of updates received from OM : " +
+ dbUpdates.getData().size());
+ for (byte[] data : dbUpdates.getData()) {
+ WriteBatch writeBatch = new WriteBatch(data);
+ writeBatch.iterate(omdbUpdatesHandler);
+ RDBBatchOperation rdbBatchOperation = new
RDBBatchOperation(writeBatch);
+ rdbBatchOperation.commit(rocksDB, new WriteOptions());
+ }
+ }
+ }
+
+ /**
+ * Based on current state of Recon's OM DB, we either get delta updates or
+ * full snapshot from Ozone Manager.
+ */
+ @VisibleForTesting
+ void syncDataFromOM() {
+ long currentSequenceNumber = getCurrentOMDBSequenceNumber();
+ boolean fullSnapshot = false;
+
+ if (currentSequenceNumber <= 0) {
+ fullSnapshot = true;
+ } else {
Review comment:
Good question @hanishakoneru . This is handled in OM side. Please check out
org.apache.hadoop.utils.db.RDBStore#getUpdatesSince.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 296708)
Time Spent: 3h 10m (was: 3h)
> Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone
> Manager.
> --------------------------------------------------------------------------------
>
> Key: HDDS-1105
> URL: https://issues.apache.org/jira/browse/HDDS-1105
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Reporter: Aravindan Vijayan
> Assignee: Aravindan Vijayan
> Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> *Some context*
> The FSCK server will periodically invoke this OM API passing in the most
> recent sequence number of its own RocksDB instance. The OM will use the
> RockDB getUpdateSince() API to answer this query. Since the getUpdateSince
> API only works against the RocksDB WAL, we have to configure OM RocksDB WAL
> (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient
> max size to make this API useful. If the OM cannot get all transactions since
> the given sequence number (due to WAL flushing), it can error out. In that
> case the FSCK server can fall back to getting the entire checkpoint snapshot
> implemented in HDDS-1085.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
