[jira] [Work logged] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=326524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326524 ] ASF GitHub Bot logged work on HDDS-1984: Author: ASF GitHub Bot Created on: 10/Oct/19 19:34 Start Date: 10/Oct/19 19:34 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#issuecomment-540746903 Thank You @arp7 and @anuengineer for review. I have committed this to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 326524) Time Spent: 3h 20m (was: 3h 10m) > Fix listBucket API > -- > > Key: HDDS-1984 > URL: https://issues.apache.org/jira/browse/HDDS-1984 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > This Jira is to fix listBucket API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listBuckets, it should use both > in-memory cache and rocksdb bucket table to list buckets in a volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=326523=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326523 ] ASF GitHub Bot logged work on HDDS-1984: Author: ASF GitHub Bot Created on: 10/Oct/19 19:34 Start Date: 10/Oct/19 19:34 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 326523) Time Spent: 3h 10m (was: 3h) > Fix listBucket API > -- > > Key: HDDS-1984 > URL: https://issues.apache.org/jira/browse/HDDS-1984 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > This Jira is to fix listBucket API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listBuckets, it should use both > in-memory cache and rocksdb bucket table to list buckets in a volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=325215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325215 ] ASF GitHub Bot logged work on HDDS-1984: Author: ASF GitHub Bot Created on: 08/Oct/19 17:57 Start Date: 08/Oct/19 17:57 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#issuecomment-539630568 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 40 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 67 | Maven dependency ordering for branch | | -1 | mvninstall | 38 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 39 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 61 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 857 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in trunk failed. | | -1 | javadoc | 20 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 959 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 34 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 38 | hadoop-ozone in the patch failed. | | -1 | compile | 26 | hadoop-hdds in the patch failed. | | -1 | compile | 20 | hadoop-ozone in the patch failed. | | -1 | javac | 26 | hadoop-hdds in the patch failed. | | -1 | javac | 20 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 57 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 714 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 28 | hadoop-hdds in the patch failed. | | -1 | unit | 26 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2468 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1555 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b7315d686214 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 91320b4 | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/6/artifact/out/patch-mvninstall-hadoop-ozone.txt | |
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=325188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325188
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 08/Oct/19 17:08
Start Date: 08/Oct/19 17:08
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1555: HDDS-1984. Fix
listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332629277
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java
##
@@ -53,4 +53,18 @@ public boolean equals(Object o) {
public int hashCode() {
return Objects.hash(key);
}
+
+ @Override
+ public int compareTo(Object o) {
+if(Objects.equals(key, ((CacheKey)o).key)) {
+ return 0;
+} else {
+ if (key instanceof String) {
+return ((String) key).compareTo((String) ((CacheKey)o).key);
+ } else {
+// If not type string, convert to string and compare.
+return key.toString().compareToCacheKey) o).key).toString());
Review comment:
Makes sense. However let's see if there is an alternate way to do this. We
should not have reflection for type-specific behavior. It is fragile.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325188)
Time Spent: 2h 50m (was: 2h 40m)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=325134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325134
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 08/Oct/19 15:44
Start Date: 08/Oct/19 15:44
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1555:
HDDS-1984. Fix listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332589314
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java
##
@@ -53,4 +53,18 @@ public boolean equals(Object o) {
public int hashCode() {
return Objects.hash(key);
}
+
+ @Override
+ public int compareTo(Object o) {
+if(Objects.equals(key, ((CacheKey)o).key)) {
+ return 0;
+} else {
+ if (key instanceof String) {
+return ((String) key).compareTo((String) ((CacheKey)o).key);
+ } else {
+// If not type string, convert to string and compare.
+return key.toString().compareToCacheKey) o).key).toString());
Review comment:
Ya, you are right, my initial approach was that. But as this Table interface
is extended by RDBTable which has byte[] as the parameter, that cannot be done.
And also, for now, this is used for Bucket and VolumeTable we should be good.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325134)
Time Spent: 2h 40m (was: 2.5h)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324817=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324817
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 08/Oct/19 02:31
Start Date: 08/Oct/19 02:31
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1555: HDDS-1984.
Fix listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332313947
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String
bucket)
}
int currentCount = 0;
-try (TableIterator>
-bucketIter = bucketTable.iterator()) {
- KeyValue kv = bucketIter.seek(startKey);
- while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) {
-kv = bucketIter.next();
-// Skip the Start Bucket if needed.
-if (kv != null && skipStartKey &&
-kv.getKey().equals(startKey)) {
+
+// For Bucket it is full cache, so we can just iterate in-memory table
+// cache.
+Iterator, CacheValue>> iterator =
Review comment:
You can consider this comment resolved. Thanks for the explanation. I am
leaving it open for other reviewers who might want to read this patch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324817)
Time Spent: 2.5h (was: 2h 20m)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324816=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324816
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 08/Oct/19 02:30
Start Date: 08/Oct/19 02:30
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1555: HDDS-1984.
Fix listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332313824
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String
bucket)
}
int currentCount = 0;
-try (TableIterator>
-bucketIter = bucketTable.iterator()) {
- KeyValue kv = bucketIter.seek(startKey);
- while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) {
-kv = bucketIter.next();
-// Skip the Start Bucket if needed.
-if (kv != null && skipStartKey &&
-kv.getKey().equals(startKey)) {
+
+// For Bucket it is full cache, so we can just iterate in-memory table
+// cache.
+Iterator, CacheValue>> iterator =
Review comment:
> But when we want to list all buckets is /vol2, we will iterate the entries
from start, and reach to /vol2 in the cache and once the maximum count is
reached we return from there.
The architecture of SkipList prevents us from iterating all the keys. That
is good enough, I was worried that we will walk all the entries. I missed we
were using a skipList based hash table.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324816)
Time Spent: 2h 20m (was: 2h 10m)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324800=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324800
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 08/Oct/19 01:44
Start Date: 08/Oct/19 01:44
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1555: HDDS-1984. Fix
listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332306087
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/TableCacheImpl.java
##
@@ -47,15 +48,22 @@
public class TableCacheImpl implements TableCache
{
- private final ConcurrentHashMap cache;
+ private final Map cache;
private final NavigableSet> epochEntries;
private ExecutorService executorService;
private CacheCleanupPolicy cleanupPolicy;
public TableCacheImpl(CacheCleanupPolicy cleanupPolicy) {
-cache = new ConcurrentHashMap<>();
+
+// As for full table cache only we need elements to be inserted in sorted
+// manner, so that list will be easy. For other we can go with Hash map.
+if (cleanupPolicy == CacheCleanupPolicy.NEVER) {
Review comment:
We do need a solution for this, it's not very good to keep adding special
checks for Cache policy NEVER. The code will become fragile.
I think when we first added cache policies you proposed making it an
interface. Let's do that now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324800)
Time Spent: 2h 10m (was: 2h)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324799
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 08/Oct/19 01:43
Start Date: 08/Oct/19 01:43
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #1555: HDDS-1984. Fix
listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332305881
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java
##
@@ -53,4 +53,18 @@ public boolean equals(Object o) {
public int hashCode() {
return Objects.hash(key);
}
+
+ @Override
+ public int compareTo(Object o) {
+if(Objects.equals(key, ((CacheKey)o).key)) {
+ return 0;
+} else {
+ if (key instanceof String) {
+return ((String) key).compareTo((String) ((CacheKey)o).key);
+ } else {
+// If not type string, convert to string and compare.
+return key.toString().compareToCacheKey) o).key).toString());
Review comment:
I think the correct fix is to make OzoneTokenIdentifier a Comparable, and
then enforce that the CacheKey implements Comparable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324799)
Time Spent: 2h (was: 1h 50m)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324700
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:45
Start Date: 07/Oct/19 23:45
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1555:
HDDS-1984. Fix listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332284284
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String
bucket)
}
int currentCount = 0;
-try (TableIterator>
-bucketIter = bucketTable.iterator()) {
- KeyValue kv = bucketIter.seek(startKey);
- while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) {
-kv = bucketIter.next();
-// Skip the Start Bucket if needed.
-if (kv != null && skipStartKey &&
-kv.getKey().equals(startKey)) {
+
+// For Bucket it is full cache, so we can just iterate in-memory table
+// cache.
+Iterator, CacheValue>> iterator =
Review comment:
To improve this API, I think we should change our cache data structure which
will be effective for both read/write/list API. (Initially, usage of
ConcurrentHashMap, as get() is a constant time operation, but it has caused the
problem for list API).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324700)
Time Spent: 1h 50m (was: 1h 40m)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324699
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:39
Start Date: 07/Oct/19 23:39
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1555:
HDDS-1984. Fix listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332282739
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String
bucket)
}
int currentCount = 0;
-try (TableIterator>
-bucketIter = bucketTable.iterator()) {
- KeyValue kv = bucketIter.seek(startKey);
- while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) {
-kv = bucketIter.next();
-// Skip the Start Bucket if needed.
-if (kv != null && skipStartKey &&
-kv.getKey().equals(startKey)) {
+
+// For Bucket it is full cache, so we can just iterate in-memory table
+// cache.
+Iterator, CacheValue>> iterator =
Review comment:
There is a maxCount, when we reach the count, we will return immediately, in
that case, we shall not iterate all volumes in a bucket.
BucketTable cache is just a concurrentHashMap of all buckets in OM.
So let's take an example: As it is ConcurrentSkipListMap, it has items
sorted based on the key.
We have entries in bucket table cache like below: (For brevity, removed
bucketInfo structures which are the values for the key.)
/vol/buck1
/vol/buck2
/vol/buck3
/vol2/bucket2
/vol2/bucket3
/vol2/bucket4
When we want to list buckets of /vol, and in each return only 1
entry(maximum count), we return /vol/buck1 and then immediately return, next
time listBuckets is called, startkey will be /vol/buck1,1, we return
/vol/buck2, (To return this we iterate 2 entries), like that so on.
But when we want to list all buckets is /vol2, we will iterate the entries
from start, and reach to /vol2 in the cache and once the maximum count is
reached we return from there.
So, to answer your question not every time we iterate entire cache map. (But
some times we iterate and skip them, as shown with case of vol2 bucket list)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324699)
Time Spent: 1h 40m (was: 1.5h)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324698
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:38
Start Date: 07/Oct/19 23:38
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1555:
HDDS-1984. Fix listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332282739
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String
bucket)
}
int currentCount = 0;
-try (TableIterator>
-bucketIter = bucketTable.iterator()) {
- KeyValue kv = bucketIter.seek(startKey);
- while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) {
-kv = bucketIter.next();
-// Skip the Start Bucket if needed.
-if (kv != null && skipStartKey &&
-kv.getKey().equals(startKey)) {
+
+// For Bucket it is full cache, so we can just iterate in-memory table
+// cache.
+Iterator, CacheValue>> iterator =
Review comment:
There is a maxCount, when we reach the count, we will return immediately, in
that case, we shall not iterate all volumes in a bucket.
BucketTable cache is just a concurrentHashMap of all buckets in OM.
So let's take an example: As it is ConcurrentSkipListMap, it has items
sorted based on the key.
We have entries in bucket table cache like below: (For brevity, removed
bucketInfo structures which are the values for the key.)
/vol/buck1
/vol/buck2
/vol/buck3
/vol2/bucket2
/vol2/bucket3
/vol2/bucket4
When we want to list buckets of /vol, and in each return only 1
entry(maximum count), we return /vol/buck1 and then immediately return, next
time listBuckets is called, startkey will be /vol/buck1,1, we return
/vol/buck2, (To return this we iterate 2 entries), like that so on.
But when we want to list all buckets is /vol2, we will iterate the entries
from start, and reach to /vol2 in the cache and once the maximum count is
reached we return from there.
So, to answer your question not every time we iterate entire cache map.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324698)
Time Spent: 1.5h (was: 1h 20m)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324696=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324696
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 07/Oct/19 23:31
Start Date: 07/Oct/19 23:31
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1555:
HDDS-1984. Fix listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332281050
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java
##
@@ -53,4 +53,18 @@ public boolean equals(Object o) {
public int hashCode() {
return Objects.hash(key);
}
+
+ @Override
+ public int compareTo(Object o) {
+if(Objects.equals(key, ((CacheKey)o).key)) {
+ return 0;
+} else {
+ if (key instanceof String) {
+return ((String) key).compareTo((String) ((CacheKey)o).key);
+ } else {
+// If not type string, convert to string and compare.
+return key.toString().compareToCacheKey) o).key).toString());
Review comment:
When CacheKey KEY type is not string. For BucketTable type is string,
for tokentable type is OzoneTokenIdentifier. As this is common class used by
all tables in OM, added the if, else condition.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324696)
Time Spent: 1h 20m (was: 1h 10m)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324663
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 07/Oct/19 22:25
Start Date: 07/Oct/19 22:25
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1555: HDDS-1984.
Fix listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332257684
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/cache/CacheKey.java
##
@@ -53,4 +53,18 @@ public boolean equals(Object o) {
public int hashCode() {
return Objects.hash(key);
}
+
+ @Override
+ public int compareTo(Object o) {
+if(Objects.equals(key, ((CacheKey)o).key)) {
+ return 0;
+} else {
+ if (key instanceof String) {
+return ((String) key).compareTo((String) ((CacheKey)o).key);
+ } else {
+// If not type string, convert to string and compare.
+return key.toString().compareToCacheKey) o).key).toString());
Review comment:
when can this happen?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324663)
Time Spent: 1h 10m (was: 1h)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[
https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=324662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324662
]
ASF GitHub Bot logged work on HDDS-1984:
Author: ASF GitHub Bot
Created on: 07/Oct/19 22:25
Start Date: 07/Oct/19 22:25
Worklog Time Spent: 10m
Work Description: anuengineer commented on pull request #1555: HDDS-1984.
Fix listBucket API.
URL: https://github.com/apache/hadoop/pull/1555#discussion_r332264370
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -618,23 +618,31 @@ public boolean isBucketEmpty(String volume, String
bucket)
}
int currentCount = 0;
-try (TableIterator>
-bucketIter = bucketTable.iterator()) {
- KeyValue kv = bucketIter.seek(startKey);
- while (currentCount < maxNumOfBuckets && bucketIter.hasNext()) {
-kv = bucketIter.next();
-// Skip the Start Bucket if needed.
-if (kv != null && skipStartKey &&
-kv.getKey().equals(startKey)) {
+
+// For Bucket it is full cache, so we can just iterate in-memory table
+// cache.
+Iterator, CacheValue>> iterator =
Review comment:
Sorry, I am not about to make sure of thi; but for each request do we
iterator through the whole bucket space here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 324662)
Time Spent: 1h 10m (was: 1h)
> Fix listBucket API
> --
>
> Key: HDDS-1984
> URL: https://issues.apache.org/jira/browse/HDDS-1984
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> This Jira is to fix listBucket API in HA code path.
> In HA, we have an in-memory cache, where we put the result to in-memory cache
> and return the response, later it will be picked by double buffer thread and
> it will flush to disk. So, now when do listBuckets, it should use both
> in-memory cache and rocksdb bucket table to list buckets in a volume.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=321631=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321631 ] ASF GitHub Bot logged work on HDDS-1984: Author: ASF GitHub Bot Created on: 02/Oct/19 00:02 Start Date: 02/Oct/19 00:02 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#issuecomment-537279761 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 34 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 34 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 16 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 49 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 846 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 23 | hadoop-hdds in trunk failed. | | -1 | javadoc | 20 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 947 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 32 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 31 | Maven dependency ordering for patch | | -1 | mvninstall | 35 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 37 | hadoop-ozone in the patch failed. | | -1 | compile | 26 | hadoop-hdds in the patch failed. | | -1 | compile | 19 | hadoop-ozone in the patch failed. | | -1 | javac | 26 | hadoop-hdds in the patch failed. | | -1 | javac | 19 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 57 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 704 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 38 | hadoop-hdds in the patch failed. | | -1 | findbugs | 27 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 28 | hadoop-hdds in the patch failed. | | -1 | unit | 28 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2410 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1555 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e8b839048240 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d947ded | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/5/artifact/out/patch-mvninstall-hadoop-ozone.txt | |
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=321073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321073 ] ASF GitHub Bot logged work on HDDS-1984: Author: ASF GitHub Bot Created on: 01/Oct/19 06:44 Start Date: 01/Oct/19 06:44 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#issuecomment-536892791 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 41 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for branch | | -1 | mvninstall | 29 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 34 | hadoop-ozone in trunk failed. | | -1 | compile | 23 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 849 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 24 | hadoop-hdds in trunk failed. | | -1 | javadoc | 16 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 946 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 32 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | -1 | mvninstall | 36 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 40 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-hdds in the patch failed. | | -1 | compile | 16 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-hdds in the patch failed. | | -1 | javac | 16 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 57 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 716 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 24 | hadoop-hdds in the patch failed. | | -1 | javadoc | 16 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 16 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 26 | hadoop-hdds in the patch failed. | | -1 | unit | 25 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2373 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1555 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1db623207a50 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 137546a | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/4/artifact/out/patch-mvninstall-hadoop-ozone.txt | |
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=321072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321072 ] ASF GitHub Bot logged work on HDDS-1984: Author: ASF GitHub Bot Created on: 01/Oct/19 06:43 Start Date: 01/Oct/19 06:43 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#issuecomment-536892631 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 45 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for branch | | -1 | mvninstall | 32 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 34 | hadoop-ozone in trunk failed. | | -1 | compile | 22 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 54 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 850 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in trunk failed. | | -1 | javadoc | 18 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 950 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 33 | hadoop-hdds in trunk failed. | | -1 | findbugs | 21 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | -1 | mvninstall | 33 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 38 | hadoop-ozone in the patch failed. | | -1 | compile | 26 | hadoop-hdds in the patch failed. | | -1 | compile | 17 | hadoop-ozone in the patch failed. | | -1 | javac | 26 | hadoop-hdds in the patch failed. | | -1 | javac | 17 | hadoop-ozone in the patch failed. | | +1 | checkstyle | 57 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 712 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 22 | hadoop-hdds in the patch failed. | | -1 | javadoc | 21 | hadoop-ozone in the patch failed. | | -1 | findbugs | 34 | hadoop-hdds in the patch failed. | | -1 | findbugs | 21 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 25 | hadoop-hdds in the patch failed. | | -1 | unit | 25 | hadoop-ozone in the patch failed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 2385 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1555 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 1142cd113314 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 137546a | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/3/artifact/out/patch-mvninstall-hadoop-ozone.txt | |
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=321037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321037 ] ASF GitHub Bot logged work on HDDS-1984: Author: ASF GitHub Bot Created on: 01/Oct/19 02:56 Start Date: 01/Oct/19 02:56 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#issuecomment-536837875 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 40 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for branch | | -1 | mvninstall | 31 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 36 | hadoop-ozone in trunk failed. | | -1 | compile | 20 | hadoop-hdds in trunk failed. | | -1 | compile | 15 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 818 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 25 | hadoop-hdds in trunk failed. | | -1 | javadoc | 16 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 919 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 34 | hadoop-hdds in trunk failed. | | -1 | findbugs | 22 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 28 | Maven dependency ordering for patch | | -1 | mvninstall | 37 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 39 | hadoop-ozone in the patch failed. | | -1 | compile | 25 | hadoop-hdds in the patch failed. | | -1 | compile | 16 | hadoop-ozone in the patch failed. | | -1 | javac | 25 | hadoop-hdds in the patch failed. | | -1 | javac | 16 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 25 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 707 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 21 | hadoop-hdds in the patch failed. | | -1 | javadoc | 20 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 20 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 30 | hadoop-hdds in the patch failed. | | -1 | unit | 28 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 2360 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1555 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 883182c3dde4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b3275ab | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/2/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall |
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=321036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321036 ] ASF GitHub Bot logged work on HDDS-1984: Author: ASF GitHub Bot Created on: 01/Oct/19 02:56 Start Date: 01/Oct/19 02:56 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555#issuecomment-536837796 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 68 | Maven dependency ordering for branch | | -1 | mvninstall | 44 | hadoop-hdds in trunk failed. | | -1 | mvninstall | 39 | hadoop-ozone in trunk failed. | | -1 | compile | 21 | hadoop-hdds in trunk failed. | | -1 | compile | 14 | hadoop-ozone in trunk failed. | | +1 | checkstyle | 62 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 851 | branch has no errors when building and testing our client artifacts. | | -1 | javadoc | 21 | hadoop-hdds in trunk failed. | | -1 | javadoc | 18 | hadoop-ozone in trunk failed. | | 0 | spotbugs | 955 | Used deprecated FindBugs config; considering switching to SpotBugs. | | -1 | findbugs | 38 | hadoop-hdds in trunk failed. | | -1 | findbugs | 22 | hadoop-ozone in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | -1 | mvninstall | 36 | hadoop-hdds in the patch failed. | | -1 | mvninstall | 40 | hadoop-ozone in the patch failed. | | -1 | compile | 26 | hadoop-hdds in the patch failed. | | -1 | compile | 18 | hadoop-ozone in the patch failed. | | -1 | javac | 26 | hadoop-hdds in the patch failed. | | -1 | javac | 18 | hadoop-ozone in the patch failed. | | -0 | checkstyle | 28 | hadoop-hdds: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 715 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 24 | hadoop-hdds in the patch failed. | | -1 | javadoc | 18 | hadoop-ozone in the patch failed. | | -1 | findbugs | 32 | hadoop-hdds in the patch failed. | | -1 | findbugs | 20 | hadoop-ozone in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 28 | hadoop-hdds in the patch failed. | | -1 | unit | 29 | hadoop-ozone in the patch failed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 2467 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1555 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 08929fca86df 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b3275ab | | Default Java | 1.8.0_222 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-mvninstall-hadoop-hdds.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-mvninstall-hadoop-ozone.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-compile-hadoop-hdds.txt | | compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-compile-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-javadoc-hadoop-hdds.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-findbugs-hadoop-hdds.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/branch-findbugs-hadoop-ozone.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1555/1/artifact/out/patch-mvninstall-hadoop-hdds.txt | | mvninstall |
[jira] [Work logged] (HDDS-1984) Fix listBucket API
[ https://issues.apache.org/jira/browse/HDDS-1984?focusedWorklogId=321027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-321027 ] ASF GitHub Bot logged work on HDDS-1984: Author: ASF GitHub Bot Created on: 01/Oct/19 02:14 Start Date: 01/Oct/19 02:14 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1555: HDDS-1984. Fix listBucket API. URL: https://github.com/apache/hadoop/pull/1555 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 321027) Remaining Estimate: 0h Time Spent: 10m > Fix listBucket API > -- > > Key: HDDS-1984 > URL: https://issues.apache.org/jira/browse/HDDS-1984 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: YiSheng Lien >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to fix listBucket API in HA code path. > In HA, we have an in-memory cache, where we put the result to in-memory cache > and return the response, later it will be picked by double buffer thread and > it will flush to disk. So, now when do listBuckets, it should use both > in-memory cache and rocksdb bucket table to list buckets in a volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
