[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=284387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-284387 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 29/Jul/19 16:39 Start Date: 29/Jul/19 16:39 Worklog Time Spent: 10m Work Description: arp7 commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-516067148 +1 The failures look unrelated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 284387) Time Spent: 4h 50m (was: 4h 40m) > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Added an RPC end point to serve the set of updates in OM RocksDB from a given > sequence number. > This will be used by Recon (HDDS-1105) to push the data to all the tasks that > will keep their aggregate data up to date. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=284388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-284388 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 29/Jul/19 16:39 Start Date: 29/Jul/19 16:39 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 284388) Time Spent: 5h (was: 4h 50m) > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 5h > Remaining Estimate: 0h > > Added an RPC end point to serve the set of updates in OM RocksDB from a given > sequence number. > This will be used by Recon (HDDS-1105) to push the data to all the tasks that > will keep their aggregate data up to date. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=283534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-283534 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 26/Jul/19 18:23 Start Date: 26/Jul/19 18:23 Worklog Time Spent: 10m Work Description: avijayanhwx commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-515553165 The test failures seem unrelated to the patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 283534) Time Spent: 4h 40m (was: 4.5h) > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Added an RPC end point to serve the set of updates in OM RocksDB from a given > sequence number. > This will be used by Recon (HDDS-1105) to push the data to all the tasks that > will keep their aggregate data up to date. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=283395=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-283395 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 26/Jul/19 14:38 Start Date: 26/Jul/19 14:38 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-515479758 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for branch | | -1 | mvninstall | 32 | hadoop-ozone in trunk failed. | | +1 | compile | 380 | trunk passed | | +1 | checkstyle | 71 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 805 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 145 | trunk passed | | 0 | spotbugs | 437 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 626 | trunk passed | | -0 | patch | 472 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 531 | the patch passed | | +1 | compile | 368 | the patch passed | | +1 | cc | 368 | the patch passed | | +1 | javac | 368 | the patch passed | | +1 | checkstyle | 72 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 666 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 66 | hadoop-hdds generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) | | +1 | findbugs | 636 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 294 | hadoop-hdds in the patch failed. | | -1 | unit | 1555 | hadoop-ozone in the patch failed. | | +1 | asflicense | 47 | The patch does not generate ASF License warnings. | | | | 6721 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestScmSafeMode | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1033 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 1f78d6aeaca6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c0a0c35 | | Default Java | 1.8.0_212 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/9/artifact/out/branch-mvninstall-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/9/artifact/out/diff-javadoc-javadoc-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/9/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/9/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/9/testReport/ | | Max. process+thread count | 5035 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/9/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282991
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 25/Jul/19 23:17
Start Date: 25/Jul/19 23:17
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1033: HDDS-1391
: Add ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r307536825
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java
##
@@ -327,6 +329,44 @@ public CodecRegistry getCodecRegistry() {
return codecRegistry;
}
+ @Override
+ public DBUpdatesWrapper getUpdatesSince(long sequenceNumber)
+ throws SequenceNumberNotFoundException {
+
+DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper();
+try {
+ TransactionLogIterator transactionLogIterator =
+ db.getUpdatesSince(sequenceNumber);
+
+ boolean flag = true;
+
+ while (transactionLogIterator.isValid()) {
+TransactionLogIterator.BatchResult result =
+transactionLogIterator.getBatch();
+long currSequenceNumber = result.sequenceNumber();
+if (flag && currSequenceNumber > 1 + sequenceNumber) {
+ throw new SequenceNumberNotFoundException("Unable to read data from"
+
+ " RocksDB wal to get delta updates. It may have already been" +
+ "flushed to SSTs.");
+}
+flag = false;
Review comment:
Explained and renamed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 282991)
Time Spent: 4h 20m (was: 4h 10m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282990
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 25/Jul/19 23:17
Start Date: 25/Jul/19 23:17
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1033: HDDS-1391
: Add ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r307536807
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java
##
@@ -327,6 +329,44 @@ public CodecRegistry getCodecRegistry() {
return codecRegistry;
}
+ @Override
+ public DBUpdatesWrapper getUpdatesSince(long sequenceNumber)
+ throws SequenceNumberNotFoundException {
+
+DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper();
+try {
+ TransactionLogIterator transactionLogIterator =
+ db.getUpdatesSince(sequenceNumber);
+
+ boolean flag = true;
+
+ while (transactionLogIterator.isValid()) {
+TransactionLogIterator.BatchResult result =
+transactionLogIterator.getBatch();
+long currSequenceNumber = result.sequenceNumber();
+if (flag && currSequenceNumber > 1 + sequenceNumber) {
+ throw new SequenceNumberNotFoundException("Unable to read data from"
+
+ " RocksDB wal to get delta updates. It may have already been" +
+ "flushed to SSTs.");
+}
+flag = false;
+if (currSequenceNumber == sequenceNumber) {
+ transactionLogIterator.next();
+ continue;
+}
+WriteBatch writeBatch = result.writeBatch();
+byte[] writeBatchData = writeBatch.data();
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 282990)
Time Spent: 4h 10m (was: 4h)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282987
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 25/Jul/19 23:13
Start Date: 25/Jul/19 23:13
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1033: HDDS-1391
: Add ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r307536091
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManager.java
##
@@ -1395,8 +1397,41 @@ public void testDBKeyMayExist() throws Exception {
RDBStore rdbStore = (RDBStore) cluster.getOzoneManager()
.getMetadataManager().getStore();
RocksDB db = rdbStore.getDb();
-UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
+OmKeyInfo keyInfo = getNewOmKeyInfo();
+OmKeyInfoCodec omKeyInfoCodec = new OmKeyInfoCodec();
+
+db.put(StringUtils.getBytesUtf16("OMKey1"),
+omKeyInfoCodec.toPersistedFormat(keyInfo));
+
+StringBuilder sb = new StringBuilder();
+Assert.assertTrue(db.keyMayExist(StringUtils.getBytesUtf16("OMKey1"),
+sb));
+Assert.assertTrue(sb.length() > 0);
+ }
+
+
+ @Test
+ public void testGetOMDBUpdates() throws IOException {
+
+DBUpdatesRequest dbUpdatesRequest =
+DBUpdatesRequest.newBuilder().setSequenceNumber(0).build();
+
+DBUpdatesWrapper dbUpdates =
+cluster.getOzoneManager().getDBUpdates(dbUpdatesRequest);
+Assert.assertTrue(dbUpdates.getData().isEmpty());
+
+//Write data to OM.
+OmKeyInfo keyInfo = getNewOmKeyInfo();
+Assert.assertNotNull(keyInfo);
+dbUpdates =
+cluster.getOzoneManager().getDBUpdates(dbUpdatesRequest);
+Assert.assertFalse(dbUpdates.getData().isEmpty());
Review comment:
Yes, that will be difficult. The only way to do is that is to write a custom
Iterator handler for RocksDB which we have done in
org.apache.hadoop.ozone.recon.tasks.OMDBUpdatesHandler. The tests in
org.apache.hadoop.ozone.recon.tasks.TestOMDBUpdatesHandler cover the data
correctness use cases.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 282987)
Time Spent: 4h (was: 3h 50m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282981 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 25/Jul/19 23:08 Start Date: 25/Jul/19 23:08 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#discussion_r307535001 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java ## @@ -199,6 +199,7 @@ public static boolean isReadOnly( case LookupFile: case ListStatus: case GetAcl: +case DBUpdates: Review comment: Yes, for now this is OK. Handling of OM HA in recon will be done later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282981) Time Spent: 3h 50m (was: 3h 40m) > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Added an RPC end point to serve the set of updates in OM RocksDB from a given > sequence number. > This will be used by Recon (HDDS-1105) to push the data to all the tasks that > will keep their aggregate data up to date. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282966
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 25/Jul/19 22:42
Start Date: 25/Jul/19 22:42
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r307529200
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManager.java
##
@@ -1395,8 +1397,41 @@ public void testDBKeyMayExist() throws Exception {
RDBStore rdbStore = (RDBStore) cluster.getOzoneManager()
.getMetadataManager().getStore();
RocksDB db = rdbStore.getDb();
-UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
+OmKeyInfo keyInfo = getNewOmKeyInfo();
+OmKeyInfoCodec omKeyInfoCodec = new OmKeyInfoCodec();
+
+db.put(StringUtils.getBytesUtf16("OMKey1"),
+omKeyInfoCodec.toPersistedFormat(keyInfo));
+
+StringBuilder sb = new StringBuilder();
+Assert.assertTrue(db.keyMayExist(StringUtils.getBytesUtf16("OMKey1"),
+sb));
+Assert.assertTrue(sb.length() > 0);
+ }
+
+
+ @Test
+ public void testGetOMDBUpdates() throws IOException {
+
+DBUpdatesRequest dbUpdatesRequest =
+DBUpdatesRequest.newBuilder().setSequenceNumber(0).build();
+
+DBUpdatesWrapper dbUpdates =
+cluster.getOzoneManager().getDBUpdates(dbUpdatesRequest);
+Assert.assertTrue(dbUpdates.getData().isEmpty());
+
+//Write data to OM.
+OmKeyInfo keyInfo = getNewOmKeyInfo();
+Assert.assertNotNull(keyInfo);
+dbUpdates =
+cluster.getOzoneManager().getDBUpdates(dbUpdatesRequest);
+Assert.assertFalse(dbUpdates.getData().isEmpty());
Review comment:
Should we unpack the update and check the contents to ensure they were
shipped correctly? Not sure if it will be straightforward to do.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 282966)
Time Spent: 3h 20m (was: 3h 10m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282967
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 25/Jul/19 22:42
Start Date: 25/Jul/19 22:42
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r307528176
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java
##
@@ -327,6 +329,44 @@ public CodecRegistry getCodecRegistry() {
return codecRegistry;
}
+ @Override
+ public DBUpdatesWrapper getUpdatesSince(long sequenceNumber)
+ throws SequenceNumberNotFoundException {
+
+DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper();
+try {
+ TransactionLogIterator transactionLogIterator =
+ db.getUpdatesSince(sequenceNumber);
+
+ boolean flag = true;
+
+ while (transactionLogIterator.isValid()) {
+TransactionLogIterator.BatchResult result =
+transactionLogIterator.getBatch();
+long currSequenceNumber = result.sequenceNumber();
+if (flag && currSequenceNumber > 1 + sequenceNumber) {
+ throw new SequenceNumberNotFoundException("Unable to read data from"
+
+ " RocksDB wal to get delta updates. It may have already been" +
+ "flushed to SSTs.");
+}
+flag = false;
Review comment:
I didn't understand why we reset this flag after the first iteration.
Perhaps it will be clear later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 282967)
Time Spent: 3h 20m (was: 3h 10m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282965
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 25/Jul/19 22:42
Start Date: 25/Jul/19 22:42
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r307527924
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java
##
@@ -327,6 +329,44 @@ public CodecRegistry getCodecRegistry() {
return codecRegistry;
}
+ @Override
+ public DBUpdatesWrapper getUpdatesSince(long sequenceNumber)
+ throws SequenceNumberNotFoundException {
+
+DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper();
+try {
+ TransactionLogIterator transactionLogIterator =
+ db.getUpdatesSince(sequenceNumber);
+
+ boolean flag = true;
Review comment:
Can you give this a more descriptive name?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 282965)
Time Spent: 3h 10m (was: 3h)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282968 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 25/Jul/19 22:42 Start Date: 25/Jul/19 22:42 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#discussion_r307528628 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java ## @@ -199,6 +199,7 @@ public static boolean isReadOnly( case LookupFile: case ListStatus: case GetAcl: +case DBUpdates: Review comment: I believe non-leader OMs will fail to service this request in HA cluster. Is that fine? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282968) Time Spent: 3.5h (was: 3h 20m) > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Added an RPC end point to serve the set of updates in OM RocksDB from a given > sequence number. > This will be used by Recon (HDDS-1105) to push the data to all the tasks that > will keep their aggregate data up to date. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282969
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 25/Jul/19 22:42
Start Date: 25/Jul/19 22:42
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r307528281
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java
##
@@ -327,6 +329,44 @@ public CodecRegistry getCodecRegistry() {
return codecRegistry;
}
+ @Override
+ public DBUpdatesWrapper getUpdatesSince(long sequenceNumber)
+ throws SequenceNumberNotFoundException {
+
+DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper();
+try {
+ TransactionLogIterator transactionLogIterator =
+ db.getUpdatesSince(sequenceNumber);
+
+ boolean flag = true;
+
+ while (transactionLogIterator.isValid()) {
+TransactionLogIterator.BatchResult result =
+transactionLogIterator.getBatch();
+long currSequenceNumber = result.sequenceNumber();
+if (flag && currSequenceNumber > 1 + sequenceNumber) {
+ throw new SequenceNumberNotFoundException("Unable to read data from"
+
+ " RocksDB wal to get delta updates. It may have already been" +
+ "flushed to SSTs.");
+}
+flag = false;
+if (currSequenceNumber == sequenceNumber) {
+ transactionLogIterator.next();
+ continue;
+}
+WriteBatch writeBatch = result.writeBatch();
+byte[] writeBatchData = writeBatch.data();
Review comment:
Nitpick: we can avoid temporary variables writeBatch and writeBatchData.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 282969)
Time Spent: 3h 40m (was: 3.5h)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282962
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 25/Jul/19 22:35
Start Date: 25/Jul/19 22:35
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r307527606
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBUpdatesWrapper.java
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Wrapper class to hold DB data read from the RocksDB log file.
+ */
+public class DBUpdatesWrapper {
+
+ private List dataList = new ArrayList<>();
+ private long currentSequenceNumber = 0;
Review comment:
Should we initialize this to -1? I am not sure if 0 can be a valid sequence
number.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 282962)
Time Spent: 3h (was: 2h 50m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282961 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 25/Jul/19 22:30 Start Date: 25/Jul/19 22:30 Worklog Time Spent: 10m Work Description: avijayanhwx commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-515238282 @arp7 Can you please review/commit this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282961) Time Spent: 2h 50m (was: 2h 40m) > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Added an RPC end point to serve the set of updates in OM RocksDB from a given > sequence number. > This will be used by Recon (HDDS-1105) to push the data to all the tasks that > will keep their aggregate data up to date. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282397=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282397 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 25/Jul/19 02:36 Start Date: 25/Jul/19 02:36 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-514871582 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 69 | Maven dependency ordering for branch | | +1 | mvninstall | 609 | trunk passed | | +1 | compile | 359 | trunk passed | | +1 | checkstyle | 75 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 861 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 158 | trunk passed | | 0 | spotbugs | 427 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 617 | trunk passed | | -0 | patch | 460 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | +1 | mvninstall | 566 | the patch passed | | +1 | compile | 371 | the patch passed | | +1 | cc | 371 | the patch passed | | +1 | javac | 371 | the patch passed | | +1 | checkstyle | 72 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 654 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 69 | hadoop-hdds generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) | | +1 | findbugs | 633 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 277 | hadoop-hdds in the patch passed. | | -1 | unit | 2023 | hadoop-ozone in the patch failed. | | +1 | asflicense | 44 | The patch does not generate ASF License warnings. | | | | 7765 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.Test2WayCommitInRatis | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.om.TestScmSafeMode | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1033 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux f5771a69a0a5 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9b8b3ac | | Default Java | 1.8.0_212 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/7/artifact/out/diff-javadoc-javadoc-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/7/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/7/testReport/ | | Max. process+thread count | 4311 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/7/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=282266=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282266 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 24/Jul/19 20:31 Start Date: 24/Jul/19 20:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-514788362 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 72 | Maven dependency ordering for branch | | +1 | mvninstall | 576 | trunk passed | | +1 | compile | 352 | trunk passed | | +1 | checkstyle | 63 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 789 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 143 | trunk passed | | 0 | spotbugs | 412 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 605 | trunk passed | | -0 | patch | 451 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 31 | Maven dependency ordering for patch | | +1 | mvninstall | 533 | the patch passed | | +1 | compile | 367 | the patch passed | | +1 | cc | 367 | the patch passed | | +1 | javac | 367 | the patch passed | | -0 | checkstyle | 32 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -0 | checkstyle | 36 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 628 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 63 | hadoop-hdds generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15) | | +1 | findbugs | 621 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 280 | hadoop-hdds in the patch passed. | | -1 | unit | 176 | hadoop-ozone in the patch failed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 5691 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.ratis.TestOzoneManagerRatisServer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1033 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux f57c9f61d1b8 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cb69700 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/diff-checkstyle-hadoop-hdds.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/diff-javadoc-javadoc-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/testReport/ | | Max. process+thread count | 1287 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=281123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-281123
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 23/Jul/19 16:08
Start Date: 23/Jul/19 16:08
Worklog Time Spent: 10m
Work Description: swagle commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r306406223
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java
##
@@ -318,6 +320,44 @@ public CodecRegistry getCodecRegistry() {
return codecRegistry;
}
+ @Override
+ public DBUpdatesWrapper getUpdatesSince(long sequenceNumber)
+ throws DataNotFoundException {
+
+DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper();
+try {
+ TransactionLogIterator transactionLogIterator =
+ db.getUpdatesSince(sequenceNumber);
+
+ boolean flag = true;
+
+ while (transactionLogIterator.isValid()) {
Review comment:
So this means we need to at least configure the flushes defensively to maybe
3 * recon sync internal. Will that be a separate Jira or is already correctly
configured?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 281123)
Time Spent: 2h 20m (was: 2h 10m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=281122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-281122 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 23/Jul/19 16:07 Start Date: 23/Jul/19 16:07 Worklog Time Spent: 10m Work Description: swagle commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-514276777 +1 LGTM, needs rebase. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 281122) Time Spent: 2h 10m (was: 2h) > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Added an RPC end point to serve the set of updates in OM RocksDB from a given > sequence number. > This will be used by Recon (HDDS-1105) to push the data to all the tasks that > will keep their aggregate data up to date. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=27=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-27 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 19/Jul/19 23:18 Start Date: 19/Jul/19 23:18 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-513409328 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 85 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for branch | | +1 | mvninstall | 509 | trunk passed | | +1 | compile | 265 | trunk passed | | +1 | checkstyle | 75 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 926 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 167 | trunk passed | | 0 | spotbugs | 357 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 572 | trunk passed | | -0 | patch | 403 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 31 | Maven dependency ordering for patch | | +1 | mvninstall | 448 | the patch passed | | +1 | compile | 268 | the patch passed | | +1 | cc | 268 | the patch passed | | +1 | javac | 268 | the patch passed | | -0 | checkstyle | 37 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -0 | checkstyle | 41 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 763 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 76 | hadoop-hdds generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) | | +1 | findbugs | 559 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 372 | hadoop-hdds in the patch passed. | | -1 | unit | 313 | hadoop-ozone in the patch failed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 5834 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.ratis.TestOzoneManagerRatisServer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1033 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 42daf491a8cd 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 7f1b76c | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/5/artifact/out/diff-checkstyle-hadoop-hdds.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/5/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/5/artifact/out/diff-javadoc-javadoc-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/5/testReport/ | | Max. process+thread count | 1304 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=279831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279831
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 19/Jul/19 16:52
Start Date: 19/Jul/19 16:52
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1033: HDDS-1391
: Add ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r305440235
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java
##
@@ -318,6 +320,44 @@ public CodecRegistry getCodecRegistry() {
return codecRegistry;
}
+ @Override
+ public DBUpdatesWrapper getUpdatesSince(long sequenceNumber)
+ throws DataNotFoundException {
+
+DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper();
+try {
+ TransactionLogIterator transactionLogIterator =
+ db.getUpdatesSince(sequenceNumber);
+
+ boolean flag = true;
+
+ while (transactionLogIterator.isValid()) {
Review comment:
yes, flushes can happen from memtables to SST anytime. If the WAL is deleted
while it is being read, we will still be handle it through a retry mechanism
from Recon side.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 279831)
Time Spent: 1h 50m (was: 1h 40m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=279830=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279830
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 19/Jul/19 16:52
Start Date: 19/Jul/19 16:52
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #1033: HDDS-1391
: Add ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r305440288
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DataNotFoundException.java
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db;
+
+import java.io.IOException;
+
+/**
+ * Thrown if RocksDB is unable to find requested data from WAL file.
+ */
+public class DataNotFoundException extends IOException {
Review comment:
Will rename this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 279830)
Time Spent: 1h 40m (was: 1.5h)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=279649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279649 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 19/Jul/19 11:11 Start Date: 19/Jul/19 11:11 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-513190749 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 0 | Docker mode activated. | | -1 | patch | 14 | https://github.com/apache/hadoop/pull/1033 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/1033 | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/4/console | | versions | git=2.7.4 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 279649) Time Spent: 1.5h (was: 1h 20m) > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Added an RPC end point to serve the set of updates in OM RocksDB from a given > sequence number. > This will be used by Recon (HDDS-1105) to push the data to all the tasks that > will keep their aggregate data up to date. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=279345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279345
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 18/Jul/19 21:08
Start Date: 18/Jul/19 21:08
Worklog Time Spent: 10m
Work Description: swagle commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r305115724
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBStore.java
##
@@ -318,6 +320,44 @@ public CodecRegistry getCodecRegistry() {
return codecRegistry;
}
+ @Override
+ public DBUpdatesWrapper getUpdatesSince(long sequenceNumber)
+ throws DataNotFoundException {
+
+DBUpdatesWrapper dbUpdatesWrapper = new DBUpdatesWrapper();
+try {
+ TransactionLogIterator transactionLogIterator =
+ db.getUpdatesSince(sequenceNumber);
+
+ boolean flag = true;
+
+ while (transactionLogIterator.isValid()) {
Review comment:
Does this imply flush to sst can happen while iterating the log?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 279345)
Time Spent: 1h 20m (was: 1h 10m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=279333=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279333
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 18/Jul/19 21:00
Start Date: 18/Jul/19 21:00
Worklog Time Spent: 10m
Work Description: swagle commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r305113051
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DataNotFoundException.java
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db;
+
+import java.io.IOException;
+
+/**
+ * Thrown if RocksDB is unable to find requested data from WAL file.
+ */
+public class DataNotFoundException extends IOException {
Review comment:
or just SequenceNumberNotFoundException?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 279333)
Time Spent: 1h 10m (was: 1h)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=279330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279330
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 18/Jul/19 20:59
Start Date: 18/Jul/19 20:59
Worklog Time Spent: 10m
Work Description: swagle commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r305112491
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DataNotFoundException.java
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db;
+
+import java.io.IOException;
+
+/**
+ * Thrown if RocksDB is unable to find requested data from WAL file.
+ */
+public class DataNotFoundException extends IOException {
Review comment:
The exception seems too broad, can be DataNotFoundFroSequenceNumberException?
The message can, therefore, ask for the sequence number and client-side code
can log it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 279330)
Time Spent: 50m (was: 40m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[
https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=279331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-279331
]
ASF GitHub Bot logged work on HDDS-1391:
Author: ASF GitHub Bot
Created on: 18/Jul/19 20:59
Start Date: 18/Jul/19 20:59
Worklog Time Spent: 10m
Work Description: swagle commented on pull request #1033: HDDS-1391 : Add
ability in OM to serve delta updates through an API.
URL: https://github.com/apache/hadoop/pull/1033#discussion_r305112491
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DataNotFoundException.java
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db;
+
+import java.io.IOException;
+
+/**
+ * Thrown if RocksDB is unable to find requested data from WAL file.
+ */
+public class DataNotFoundException extends IOException {
Review comment:
The exception seems too broad, can be DataNotFoundForSequenceNumberException?
The message can, therefore, ask for the sequence number and client-side code
can log it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 279331)
Time Spent: 1h (was: 50m)
> Add ability in OM to serve delta updates through an API.
>
>
> Key: HDDS-1391
> URL: https://issues.apache.org/jira/browse/HDDS-1391
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Added an RPC end point to serve the set of updates in OM RocksDB from a given
> sequence number.
> This will be used by Recon (HDDS-1105) to push the data to all the tasks that
> will keep their aggregate data up to date.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=278521=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-278521 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 17/Jul/19 22:08 Start Date: 17/Jul/19 22:08 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-512587140 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 44 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for branch | | +1 | mvninstall | 547 | trunk passed | | +1 | compile | 256 | trunk passed | | +1 | checkstyle | 59 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 780 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 150 | trunk passed | | 0 | spotbugs | 307 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 598 | trunk passed | | -0 | patch | 344 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | +1 | mvninstall | 423 | the patch passed | | +1 | compile | 268 | the patch passed | | +1 | cc | 268 | the patch passed | | +1 | javac | 268 | the patch passed | | +1 | checkstyle | 81 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 648 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 70 | hadoop-hdds generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) | | +1 | findbugs | 522 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 300 | hadoop-hdds in the patch failed. | | -1 | unit | 153 | hadoop-ozone in the patch failed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 5251 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.container.placement.algorithms.TestContainerPlacementFactory | | | hadoop.ozone.om.ratis.TestOzoneManagerRatisServer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1033 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 5640f7cf3c34 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 303a7f8 | | Default Java | 1.8.0_212 | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/3/artifact/out/diff-javadoc-javadoc-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/3/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/3/testReport/ | | Max. process+thread count | 1401 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 278521) Time Spent: 40m (was: 0.5h) > Add ability in OM to serve delta updates through an API. >
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=269627=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-269627 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 28/Jun/19 23:35 Start Date: 28/Jun/19 23:35 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-506905650 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 52 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for branch | | +1 | mvninstall | 500 | trunk passed | | +1 | compile | 257 | trunk passed | | +1 | checkstyle | 74 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 869 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 164 | trunk passed | | 0 | spotbugs | 311 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 498 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 37 | Maven dependency ordering for patch | | +1 | mvninstall | 453 | the patch passed | | +1 | compile | 269 | the patch passed | | +1 | cc | 269 | the patch passed | | +1 | javac | 269 | the patch passed | | -0 | checkstyle | 41 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 687 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 70 | hadoop-hdds generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) | | +1 | findbugs | 518 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 108 | hadoop-hdds in the patch failed. | | -1 | unit | 209 | hadoop-ozone in the patch failed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 5173 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.utils.db.TestRDBStore | | | hadoop.ozone.om.ratis.TestOzoneManagerRatisServer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1033 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 6169ac17e161 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 29465bf | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/1/artifact/out/diff-javadoc-javadoc-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/1/testReport/ | | Max. process+thread count | 1401 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1033/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 269627) Time Spent: 0.5h (was: 20m) > Add ability in OM to serve delta updates through an API. >
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=269584=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-269584 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 28/Jun/19 22:20 Start Date: 28/Jun/19 22:20 Worklog Time Spent: 10m Work Description: avijayanhwx commented on issue #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033#issuecomment-506893550 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 269584) Time Spent: 20m (was: 10m) > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Added an RPC end point to serve the set of updates in OM RocksDB from a given > sequence number. > This will be used by Recon (HDDS-1105) to push the data to all the tasks that > will keep their aggregate data up to date. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1391) Add ability in OM to serve delta updates through an API.
[ https://issues.apache.org/jira/browse/HDDS-1391?focusedWorklogId=269568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-269568 ] ASF GitHub Bot logged work on HDDS-1391: Author: ASF GitHub Bot Created on: 28/Jun/19 22:08 Start Date: 28/Jun/19 22:08 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #1033: HDDS-1391 : Add ability in OM to serve delta updates through an API. URL: https://github.com/apache/hadoop/pull/1033 Added an RPC end point to serve the set of updates in OM RocksDB from a given sequence number. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 269568) Time Spent: 10m Remaining Estimate: 0h > Add ability in OM to serve delta updates through an API. > > > Key: HDDS-1391 > URL: https://issues.apache.org/jira/browse/HDDS-1391 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
