[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455100#comment-16455100 ] Hudson commented on HDFS-11779: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14070/]) HDFS-11779. Ozone: KSM: add listBuckets. Contributed by Weiwei Yang. (omalley: rev 0c37e05b10b6793522c11d5955f96196c8cd26b2) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeySpaceManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/KeySpaceManagerProtocol.proto * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBStore.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/MetadataManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ksm/protocolPB/KeySpaceManagerProtocolClientSideTranslatorPB.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ksm/protocol/KeySpaceManagerProtocol.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/MetadataManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/storage/DistributedStorageHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/ksm/TestKeySpaceManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocolPB/KeySpaceManagerProtocolServerSideTranslatorPB.java > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch, > HDFS-11779-HDFS-7240.014.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450695#comment-16450695 ] Hudson commented on HDFS-11779: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) HDFS-11779. Ozone: KSM: add listBuckets. Contributed by Weiwei Yang. (wwei: rev 23d7d613df5f7663b955c531eae01d5a287df379) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/MetadataManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ksm/protocolPB/KeySpaceManagerProtocolClientSideTranslatorPB.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeySpaceManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/KeySpaceManagerProtocol.proto * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/MetadataManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/ksm/TestKeySpaceManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocolPB/KeySpaceManagerProtocolServerSideTranslatorPB.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/storage/DistributedStorageHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBStore.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ksm/protocol/KeySpaceManagerProtocol.java > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch, > HDFS-11779-HDFS-7240.014.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042475#comment-16042475 ] Weiwei Yang commented on HDFS-11779: Committed to the feature branch. Thanks [~anu], [~xyao], [~linyiqun], [~nandakumar131], [~yuanbo] for the reviews. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Fix For: HDFS-7240 > > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch, > HDFS-11779-HDFS-7240.014.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042399#comment-16042399
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
34s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
52s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.scm.TestContainerSQLCli |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12872002/HDFS-11779-HDFS-7240.014.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 396e9d4a4a93 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / da8bc38 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042371#comment-16042371
]
Weiwei Yang commented on HDFS-11779:
Yeah, you are right. I think you are talking about a client side pagination
implementation based on rest call. That could happen, as listBuckets in
{{MetadataManagerImpl}} only holds a read lock. So in the case you mentioned,
step 3 will fail with key not found error. Client side will need to handle this
and decide how to proceed, maybe throw an inconsistent state exception. I will
file a JIRA to track this. Thank you.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch,
> HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch,
> HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch,
> HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch,
> HDFS-11779-HDFS-7240.014.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042356#comment-16042356 ] Anu Engineer commented on HDFS-11779: - Sorry, I was not very clear. I was thinking about the case where the list is happening and the continue call is made with the startKey. But if the startKey to continue has been deleted, we need to handle it somehow or the client needs to tell us how to continue. This is a corner case, but we had discussed that in the list case. Even if you have snapshot taken, it is still possible to get into that situation. Here is the an example situation 1. A list call is made say from key1 to key100 is returned. 2. Someone deletes the key100 3. A list continue call is made with startKey=key100. If this is handled already may be I missed it, if not it might be a good idea to file a JIRA to track this so we don't forget it. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch, > HDFS-11779-HDFS-7240.014.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042324#comment-16042324
]
Weiwei Yang commented on HDFS-11779:
Hi [~anu] thanks. Can you be more specific what the case you want to track? The
patch allows the case while startKey is missing,
{code}
@param startBucket
Optional start bucket name parameter indicating where to start
the bucket listing from.
{code}
please also see test case
{code}
// the 3rd argument is the startKey
listBucketArgs = new ListArgs(volArgs, null, 100, null);
result = storageHandler.listBuckets(listBucketArgs);
Assert.assertEquals(20, result.getBuckets().size());
{code}
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch,
> HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch,
> HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch,
> HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch,
> HDFS-11779-HDFS-7240.014.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042308#comment-16042308 ] Anu Engineer commented on HDFS-11779: - +1, only one minor comment. It would be a good idea to file a JIRA to track how we will handle missing startKey. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch, > HDFS-11779-HDFS-7240.014.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042280#comment-16042280 ] Anu Engineer commented on HDFS-11779: - Thanks, Looking at it right now > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch, > HDFS-11779-HDFS-7240.014.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042278#comment-16042278 ] Weiwei Yang commented on HDFS-11779: Rebased done, v14 patch uploaded. Thanks. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch, > HDFS-11779-HDFS-7240.014.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042258#comment-16042258 ] Weiwei Yang commented on HDFS-11779: Thanks [~anu], conflicts are caused by HDFS-11880, let me rebase it to latest code base and upload a new patch. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042250#comment-16042250 ] Anu Engineer commented on HDFS-11779: - [~cheersyang] I am not able to apply this patch. Would you please verify this patch on your end. I will be online for some time more in case you are uploading a new patch. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040733#comment-16040733
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
36s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
52s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
13s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.ozone.scm.TestContainerSQLCli |
| Timed out junit tests |
org.apache.hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | org.apache.hadoop.hdfs.TestMultiThreadedHflush |
| | org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | org.apache.hadoop.hdfs.TestGetBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12871807/HDFS-11779-HDFS-7240.013.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 5b76813f17af 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040605#comment-16040605 ] Weiwei Yang commented on HDFS-11779: Posted v13 patch based on the work from HDFS-11926, please kindly review. Thanks > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch, HDFS-11779-HDFS-7240.013.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037172#comment-16037172 ] Weiwei Yang commented on HDFS-11779: Pause the progress here, as I found all *LIST* functions required a common utility function to retrieve a range of KVs from levelDB. I have raised HDFS-11926 to track this. I will resume the progress once that is done. So other tasks can move on. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch, > HDFS-11779-HDFS-7240.012.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036774#comment-16036774
]
Yuanbo Liu commented on HDFS-11779:
---
[~cheersyang] Just some minor comments:
1. MetadataManagerImpl.java
{quote}
if (Strings.isNullOrEmpty(volumeName)) {
return result;
}
byte[] volumeNameBytes = getVolumeKey(volumeName);
if (store.get(volumeNameBytes) == null) {
throw new KSMException("Volume " + volumeName + " not found.",
ResultCodes.FAILED_VOLUME_NOT_FOUND);
}
{quote}
When volumeName is null, it returns an empty bucket list, when volumeName
cannot be found,
it throws an exception. I'm wondering whether it's better that these two cases
have the same behavior, either returning empty
list or throwing exception.
2. re-base the patch please.
Others look good to me.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch,
> HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch,
> HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch,
> HDFS-11779-HDFS-7240.012.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035176#comment-16035176
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
7s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 1
unchanged - 1 fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
28s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}120m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870994/HDFS-11779-HDFS-7240.012.patch
|
| Optional Tests | asflicense compile javac javadoc
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034131#comment-16034131
]
Weiwei Yang commented on HDFS-11779:
UT failure in class {{TestKeySpaceManager}} was not related to this patch, I
have created HDFS-11913 to fix that.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch,
> HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch,
> HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034079#comment-16034079
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
35s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
51s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 1
unchanged - 1 fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 20s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.ozone.ksm.TestKeySpaceManager |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.cblock.TestCBlockServer |
| Timed out junit tests |
org.apache.hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
|
| | org.apache.hadoop.hdfs.TestGetBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034012#comment-16034012 ] Weiwei Yang commented on HDFS-11779: Rebased to latest code base. [~xyao], [~anu], [~linyiqun], [~nandakumar131], [~yuanbo], do you have any more comments to the latest patch? Please let me know. Thanks > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch, HDFS-11779-HDFS-7240.011.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032417#comment-16032417
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
56s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
1 unchanged - 1 fixed = 2 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
|
| | hadoop.cblock.TestCBlockCLI |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.cblock.TestCBlockServerPersistence |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| Timed out junit tests | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870711/HDFS-11779-HDFS-7240.010.patch
|
|
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032313#comment-16032313 ] Weiwei Yang commented on HDFS-11779: Rebased to latest code base in v10 patch. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch, > HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch, > HDFS-11779-HDFS-7240.010.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030992#comment-16030992
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
17s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870539/HDFS-11779-HDFS-7240.009.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux a83866cca3f3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 122d660 |
| Default Java | 1.8.0_131 |
|
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030989#comment-16030989
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
9s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.cblock.TestCBlockServer |
| | hadoop.ozone.scm.TestContainerSQLCli |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| Timed out junit tests | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870539/HDFS-11779-HDFS-7240.009.patch
|
| Optional Tests | asflicense
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030843#comment-16030843
]
Weiwei Yang commented on HDFS-11779:
Thanks [~yuanbo] for the comments. I have fixed 2.a, 2.b in my v9 patch,
additionally I added an utility class {{LevelDBStore#StringKeyPrefixFilter}}
for better re-use. Regarding to your comment 1, thanks for pointing that out.
It makes sense to me. Right now {{FilteredKeys}} class lives in
{{org.apache.hadoop.ozone.container.common.helpers}} which is not common for
all layers, I think we need another JIRA to refactor that code to call
{{LevelDBStore#getRangeKVs}} with {{LevelDBKeyFilter}}, [~anu] what do you
think?
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch,
> HDFS-11779-HDFS-7240.008.patch, HDFS-11779-HDFS-7240.009.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030758#comment-16030758
]
Yuanbo Liu commented on HDFS-11779:
---
[~cheersyang]Thanks for v8 patch, here are my comments:
1. LevelDBKeyFilter.java.
There is a class file called {{FilteredKeys.java}} in ozone project, can we
leverage this part of work and combine them together
so that we can get rid of some overlap code.
2. LevelDBStore.java
a.
{code}
result.add(new AbstractMap.SimpleEntry(
current.getKey(), current.getValue()));
{code}
we can add the entry directly here like this, there seems no need to copy-add
because we read the value from DB(Native code)
{code}
result.add(current)
{code}
b.
{code}
DBIterator dbIter = getIterator();
{code}
Since DBIterator is a closeable class, we can use {{try}} to wrap this code
sentence for safely closing {{dbIter}}
{code}
try(DBIterator dbIter = getIterator()) {
...
}
{code}
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch,
> HDFS-11779-HDFS-7240.008.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030721#comment-16030721
]
Weiwei Yang commented on HDFS-11779:
Thanks [~linyiqun] and [~nandakumar131] for the comments, I have uploaded v8
patch with following changes
Comment from [~linyiqun]
bq. The maxNumOfKeys < 0 should be maxNumOfKeys <=0, var maxNumOfKeys should be
a positive number.
It makes sense. Allow {{maxNumOfKeys = 0}} seems unnecessary, I have updated
the patch so it only takes value *(0, MAX_LISTBUCKETS_SIZE]*.
Comments from [~nandakumar131]
bq. Can we use getVolumeKey(volumeName) as prevKey in
MetadataManagerImpl#listBuckets, so that we can skip iterating over the other
previous volumes?
Good catch. I agree, that improves the performance. I have updated the patch to
do so. The change is done in {{MetadataManagerImpl#listBuckets}}.
bq. So even if the startKey is not valid, we will proceed with the next valid
key. We are returning an empty list only if the filters doesn't match.
If startKey is not valid, once done {{iterator.seek(key)}}, the iterator will
not return any item in next iteration. The doc is correct. You can refer to the
test case in {{TestKeySpaceManager}}
{code}
// Provide an invalid bucket name as start key.
listBucketArgs = new ListArgs(volArgs, null, 100, "unknown_bucket_name");
result = storageHandler.listBuckets(listBucketArgs);
Assert.assertEquals(0, result.getBuckets().size());
{code}
Thanks
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch,
> HDFS-11779-HDFS-7240.008.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029206#comment-16029206
]
Nandakumar commented on HDFS-11779:
---
Thanks for the updated patch [~cheersyang].
Please correct me, if my understanding is wrong
While making the first call of listBucket the client doesn't have any knowledge
on prevBucket, so this value will be null.
In that case {{prevKey}} in {{MetadataManagerImpl#listBuckets}} is also null
{code}
if (!Strings.isNullOrEmpty(prevBucket)) {
prevKey = getBucketKey(volumeName, prevBucket);
}
{code}
So the {{startKey}} of {{LevelDBStore#getRangeKVs}} is also null, and the
iteration will start from beginning of level DB
{code}
if (startKey != null) {
dbIter.seek(startKey);
} else {
dbIter.seekToFirst();
}
{code}
Can we use {{getVolumeKey(volumeName)}} as {{prevKey}} in
{{MetadataManagerImpl#listBuckets}}, so that we can skip iterating over the
other previous volumes?
NIT:
According to javadoc of LevelDBStore#getRangeKVs {{If the given startKey is not
valid, it returns an empty list}}.
This is not true, according to javadoc of DBIterator#seek {{Repositions the
iterator so the key of the next BlockElement returned greater than or equal to
the specified targetKey}}.
So even if the startKey is not valid, we will proceed with the next valid key.
We are returning an empty list only if the filters doesn't match.
The javadoc can be corrected here.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028395#comment-16028395
]
Yiqun Lin commented on HDFS-11779:
--
Thanks [~cheersyang] for updated patch. Just catch one nit in latest patch:
{code}
+ int maxNumOfKeys = args.getMaxKeys();
+ if (maxNumOfKeys < 0 ||
+ maxNumOfKeys > OzoneConsts.MAX_LISTBUCKETS_SIZE) {
+throw new IllegalArgumentException(
+String.format("Illegal max number of keys specified,"
++ " the value must be in range (0, %d), actual : %d.",
+OzoneConsts.MAX_LISTBUCKETS_SIZE, maxNumOfKeys));
+ }
{code}
The {{maxNumOfKeys < 0}} should be {{maxNumOfKeys <=0 }}, var {{maxNumOfKeys}}
should be a positive number.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch, HDFS-11779-HDFS-7240.007.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027580#comment-16027580
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
57s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 2s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| Timed out junit tests | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870216/HDFS-11779-HDFS-7240.007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 16b1c3260c26
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027511#comment-16027511
]
Weiwei Yang commented on HDFS-11779:
Hello [~xyao] and [~anu]
V7 patch contains following changes
bq. Nullable and Nonnull annotations
Removed from v7 patch.
bq. BucketManagerImpl.java: Check if this value is greater than 0 ?
maxNumOfBuckets
Add a check in {{DistributedStorageHandler#listBuckets}} to check make sure the
{{maxKey}} value is in range (0, 1024), the max value defined in
{{OzoneConsts#MAX_LISTBUCKETS_SIZE}}.
bq. KeySpaceManager.java: catch (Exception ex) ==> should this be IOException
Fixed.
bq. MetadataManagerImpl.java: Just a meta-comment – getVolumeBucket I don't
want to change this code in this patch, but I would like to formalize all these
parsing assumptions in one single class.
Removed {{getVolumeBucket}}, it is no longer needed in latest patch.
bq. Would you not be able to collapse this into a single filter.
Yes, I've done this in latest patch in a single filter.
bq. DistributedStorageHandler.java Line 272: To avoid overflow, we might
consider enforce paging by having a default MaxKeys per list request (say 100)
if the MaxKeys is not specified in the request. If the caller really want to
listAll, they can specify the MaxKeys with a negative value (like -1).
Per Anu's comment, I have added a check to make sure a request can only
retrieve 0 ~ 1024 number of buckets per request. End user is not able to
retrieve all buckets once by providing a negative number. This should be enough
for user and also safer.
bq. ResultCodes.FAILED_VOLUME_NOT_FOUND ==> VOLUME_NOT_FOUND
This is the only one comment I did not address in latest patch. Via HDFS-11769,
{{KSMException}} was added to KSM to wrap up server side exceptions, and
{{KeySpaceManagerProtocolServerSideTranslatorPB}} convert the exception into
{{ListBucketsResponse}} which contains a {{Status}} info. Then
{{KeySpaceManagerProtocolServerClientTranslatorPB}} parses the {{Status}} info
and throws an IOException. If we want to change this manner, there is a lot of
places to change. Looks like this needs some more work to get it consistent
every where.
Thanks
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027157#comment-16027157 ] Anu Engineer commented on HDFS-11779: - bq. javax.annotation because I found many other places in HDFS project used this annotation and this helps to read the function signature so I added them. If you insist I can remove this in next patch. I am personally conflicted with this. I know it is a great annotation to have and Java 8 has support for checkers. Unfortunately most of these annotations live in some checker packages -- take a look at FindBugs Documentation on checkers. That said, I am fine with this change if you think it is ok to use. It is just that javax is the cross-platform UI package. If we ever decide to create layers like java 9 -- these will come back to haunt us. So right now I leave it to your judgement. bq. maxNumOfBuckets, because it allows a value <0, which means no limit at all. While it is a good idea to have no limit support from user perspective, it has the potential for abuse. So I think we must not only check for the negative value, but also define a max entries in a listing operation( like [~xyao] is suggesting -- may be a max value of 1024) . Since we support paging, I don't see why a user will want to read arbitrarily large listing from ozone. I am not able to convince myself that we should give such an option to user. bq. I intent to raise another JIRA to improve this. Perfect, that is my intention as well > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027152#comment-16027152
]
Weiwei Yang commented on HDFS-11779:
Hello [~anu] and [~xyao]
Thanks for your comments, I will submit a new patch today to address them. Few
things though
bq. Nullable and Nonnull annotations
I used {{javax.annotation}} because I found many other places in HDFS project
used this annotation and this helps to read the function signature so I added
them. If you insist I can remove this in next patch. Thanks.
bq. BucketManagerImpl.java: Check if this value is greater than 0 ?
maxNumOfBuckets
This is the server side impl class, I don't think we need to check the value of
{{maxNumOfBuckets}}, because it allows a value <0, which means no limit at all.
bq. Just a meta-comment – getVolumeBucket I don't want to change this code in
this patch, but I would like to formalize all these parsing assumptions in one
single class – so this info is localized in one file.
I intent to raise another JIRA to improve this. Right now the key in KSM might
be in multiple schemas, 1) volume key - "/volume_name"; 2) bucket key -
"/volume_name/bucket_name"; 3) user key - "$user_name". I would like to create
a protobuf message to encapsulate this, so we can leverage protobuf to parse
and get byte arrays then persist them into KSM DB. If you agree, I will work on
this on another JIRA.
bq. Line 280-287: this can be simplified with
bucketsInfo.add(bucketInfo.getProtobuf());
The problem is the BucketInfo in {{ListBuckets}} is
{{org.apache.hadoop.ozone.web.response.BucketInfo}}, but
bucketInfo.getProtobuf() returns {{KeySpaceManagerProtocolProtos.BucketInfo}}.
So same name but different package.
Thanks
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026980#comment-16026980
]
Xiaoyu Yao commented on HDFS-11779:
---
Thanks [~cheersyang] for the patch. The v6 patch looks excellent to me. Just
have few minor issues in addition to [~anu]'s comments above.
DistributedStorageHandler.java
Line 272: To avoid overflow, we might consider enforce paging by having a
default MaxKeys per list request (say 100) if the MaxKeys is not specified in
the request. If the caller really want to listAll, they can specify the MaxKeys
with a negative value (like -1).
Line 280-287: this can be simplified with
{code} bucketsInfo.add(bucketInfo.getProtobuf()); {code}
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026529#comment-16026529
]
Anu Engineer commented on HDFS-11779:
-
[~cheersyang] Thanks for the patch. My comments are based on the earlier patch,
but I thought they are still relevant. Some very minor comments.
* {{KeySpaceManagerProtocol.java}} and few other files.
{code}
import javax.annotation.Nullable;
import javax.annotation.Nonnull;
{code}
Personally I like these annotations, but javax is traditionally a namespace
that is used for UI. While I have no better suggestion (I know there are other
projects which can provide this), would you
please reconsider if we need to add these ?
* {{BucketManagerImpl.java}}
Check if this value is greater than 0 ? _maxNumOfBuckets_
* {{KeySpaceManager.java}}
catch (Exception ex) ==> should this be IOException
* {{MetadataManagerImpl.java}}
Just a meta-comment -- {{getVolumeBucket}} I don't want to change this code in
this patch, but I would
like to formalize all these parsing assumptions in one single class -- so this
info is localized in one file.
* {{ResultCodes.FAILED_VOLUME_NOT_FOUND}} ==> VOLUME_NOT_FOUND
* Please forgive me if my understanding is not correct. Would you not be able
to collapse this into a single
filter -- since /volume/prefix -- must match -- and both levelDB and RocksDB
would do that match ?
{code}
246 LevelDBKeyFilter bucketPrefixFilter = key -> {
247 String[] volumeBucket = getVolumeBucket(key);
248 return bucketPrefix == null ? true :
249 volumeBucket[1].startsWith(bucketPrefix);
250 };
251 LevelDBKeyFilter volumeNameFilter = key -> {
252 String[] volumeBucket = getVolumeBucket(key);
253 return volumeName.equals(volumeBucket[0]);
254 };
{code}
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch,
> HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch,
> HDFS-11779-HDFS-7240.006.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026303#comment-16026303 ] Weiwei Yang commented on HDFS-11779: [~xyao], [~nandakumar131] and [~linyiqun], please see if latest patch looks good. Thanks. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch, > HDFS-11779-HDFS-7240.004.patch, HDFS-11779-HDFS-7240.005.patch, > HDFS-11779-HDFS-7240.006.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026135#comment-16026135
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
57s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870022/HDFS-11779-HDFS-7240.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux e817b09b9d02 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / e641bee |
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025947#comment-16025947
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
30s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
44s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
31s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
54s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 54s{color} |
{color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 54s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12870016/HDFS-11779-HDFS-7240.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 271384696b99 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / e641bee |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025883#comment-16025883
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
37s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
23s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869993/HDFS-11779-HDFS-7240.004.patch
|
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025774#comment-16025774 ] Weiwei Yang commented on HDFS-11779: Hi [~linyiqun] Thank you for reviewing the patch, looking forward for more comments if you have. Thanks. bq. Can you use numBuckets to replace the parameter count? Fixed. bq. It seems we don't need to print test directory here, right? Correct, removed that line. bq. Can you add more detailed error info when we found the volume bucket is null. I don't think this is an error here. The KSM database stores not only bucket key (with schema /volume_name/bucket_name), it also stores volume key (with schema /volume_name) and user key (with schema $user_name). It is expected that some of the keys will not match the pattern here so null value is expected instead of an error. Please let me know if this makes sense to you. Thank you. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch, HDFS-11779-HDFS-7240.003.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025689#comment-16025689
]
Yiqun Lin commented on HDFS-11779:
--
Thanks [~cheersyang] for working on this. I just take a quick look now, and
caught some nits:
* Can you use {{numBuckets}} to replace the parameter {{count}}? It will be
good to understood.
{code}
+ public List listBuckets(String volumeName,
+ String startKey, String prefix, int count) throws IOException {
+try {
+ metrics.incNumBucketLists();
+ return bucketManager.listBuckets(volumeName, startKey, prefix, count);
+} catch (Exception ex) {
+ metrics.incNumBucketListFails();
+ throw ex;
+}
{code}
* It seems we don't need to print test directory here, right?
{code}
+public class TestLevelDBStore {
+
+ private LevelDBStore store;
+ private File testDir;
+
+ @Before
+ public void init() throws IOException {
+testDir = GenericTestUtils.getTestDir(getClass().getSimpleName());
+System.out.println(testDir.getAbsolutePath()); <==
+store = new LevelDBStore(testDir, true);
+ }
{code}
* Can you add more detailed error info when we found the volume bucket is null.
{code}
+ public String[] getVolumeBucket(byte[] bytes) {
+String[] volumeBucket = new String[2];
+String bucketKey = DFSUtil.bytes2String(bytes);
+Path p = null;
+try {
+ p = new Path(bucketKey);
+} catch (IllegalArgumentException e) {
+ return null; <
+}
+// looking for "/volume_name/bucket_name" schema
+if (p.isRoot()) {
+ return null; <
+}else if (p.getParent().isRoot()) {
+ return null; <
+} else if (!p.getParent().getParent().isRoot()) {
+ return null; <
+} else {
+ volumeBucket[0] = p.getParent().getName();
+ volumeBucket[1] = p.getName();
+ return volumeBucket;
+}
+ }
{code}
Will give more detailed review comments soon.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch,
> HDFS-11779-HDFS-7240.002.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025461#comment-16025461
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
34s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
47s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
46s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 2 new + 9
unchanged - 0 fixed = 11 total (was 9) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.ksm.TestKeySpaceManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869916/HDFS-11779-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux dc1429d98508 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025265#comment-16025265 ] Weiwei Yang commented on HDFS-11779: V2 patch attached, addressed [~nandakumar131]'s comments and fixed some checkstyle issues. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch, > HDFS-11779-HDFS-7240.002.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025170#comment-16025170
]
Hadoop QA commented on HDFS-11779:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
58s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
36s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
47s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project: The patch generated 6 new +
0 unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
45s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 2 new + 9
unchanged - 0 fixed = 11 total (was 9) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
23s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 20s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.ksm.TestKeySpaceManager |
| | hadoop.ozone.scm.TestContainerSQLCli |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11779 |
| JIRA
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025152#comment-16025152 ] Nandakumar commented on HDFS-11779: --- Thanks [~cheersyang] for the quick update. The problem is that we are not only iterating over all the buckets of that particular volume, but all the volumes (and its buckets) which comes after that in the levelDB store. We surely can track this in another jira. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11779-HDFS-7240.001.patch > > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025138#comment-16025138
]
Weiwei Yang commented on HDFS-11779:
Hi [~nandakumar131]
Thanks for reviewing this. I agree with your all 3 comments, I'll address #1
and #2 in next patch, about #3, I listed as a TODO item and I plan to do that
in another patch. See {{DistributedStorageHandler#listBuckets}}
{noformat}
// Get bucket list from KSM. Note it is possible the client
// only provides the volume name. KSM will return all buckets
// belong to this user in this volume in a huge list.
// TODO add a safe check in case bucket list overflows, e.g >100.
{noformat}
but yeah, it probably worth to add a check for the utility method
{{LevelDBStore#getRangeKVs}} as well, but it would be tricky how to define the
limit... anyway, I will open another JIRA to track this.
Thank you.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025121#comment-16025121
]
Nandakumar commented on HDFS-11779:
---
Thanks [~cheersyang] for working on this. Please find my comments below.
1. With current logic listBuckets will return empty list even if the Volume is
not present, I think we should throw {{KSMException#FAILED_VOLUME_NOT_FOUND}}.
If we return an empty list the user might think that there is a volume with no
buckets.
2. In {{MetadataManagerImpl#listBuckets}} volumeNameFilter actually checks for
bucketPrefix, we can rename it to bucketPrefixFilter; similarly prevKeyFilter
checks for volumeName, this can be renamed to volumeNameFilter
3. {{LevelDBStore#getRangeKVs}} doesn't have endKey, so it will iterate till
the end. When the database is huge and we are querying for a volume which is
present on top, even though the result is restricted using filters, we will
unnecessary be iterating over the whole database.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024972#comment-16024972
]
Weiwei Yang commented on HDFS-11779:
V1 patch attached, please kindly review. Some notes for reviewers
1. Please read {{TestKeySpaceManager#testListBuckets}} to see how
{{listBuckets}} api is used. The volume name is required, rest of arguments are
optional. I tried to include as many as combinations when specifying arguments.
2. To be able to query buckets under constrain of {{ListArgs}} arguments, I
have created a helper method in {{LevelDBStore}}, which is
{code}
public List> getRangeKVs(@Nullable byte[] startKey,
int count, LevelDBKeyFilter... filters) throws IOException
{code}
this helps to get a range of entries from DB based on the conditions. It should
be able to be reused in other api implementation such as listVolumes, or any
other one involves to search a range of keys. I have created a test class to
test this api, see {{TestLevelDBStore#testGetRangeKVs}}.
Hope this helps. Thanks.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
> Attachments: HDFS-11779-HDFS-7240.001.patch
>
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[ https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024291#comment-16024291 ] Weiwei Yang commented on HDFS-11779: Thanks [~xyao] for the quick reply, I am fine with that. Lets stick to the design then. > Ozone: KSM: add listBuckets > --- > > Key: HDFS-11779 > URL: https://issues.apache.org/jira/browse/HDFS-11779 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > > Lists buckets of a given volume. Similar to listVolumes, paging supported via > prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024276#comment-16024276
]
Xiaoyu Yao commented on HDFS-11779:
---
Thanks [~cheersyang] for picking up this work. According to the design spec,
Ozone REST API for volume list and bucket list are all defined and with prefix
based filtering and paging support. In the latest KSM spec, we don't have a
restriction on the number of buckets per volume. It is good to have paging and
filtering to scale for large number of buckets.
The test implementation OzoneMetadataManger#listBuckets does have a TODO for
that but I think we should implement that for KSM.
{code}// TODO : Query using Prefix and PrevKey {code}
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11779) Ozone: KSM: add listBuckets
[
https://issues.apache.org/jira/browse/HDFS-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024219#comment-16024219
]
Weiwei Yang commented on HDFS-11779:
I studied amazone s3 a bit, according to s3 document
[http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#listBuckets--],
it simply returns a list of buckets.
{code}
public List listBuckets() throws SdkClientException,
AmazonServiceException
{code}
Do we really to support pagination for buckets? There won't be too many buckets
in a volume, I propose to use similar signature as s3.
{code}
/**
* Returns a list of all buckets in the given volume.
*
* @param volumeName name of the volume.
* @return a list of buckets.
* @throws IOException
*/
public List listBuckets(String volumeName) throws IOException;
{code}
[~xyao], [~anu], does this make sense to you?
Thank you.
> Ozone: KSM: add listBuckets
> ---
>
> Key: HDFS-11779
> URL: https://issues.apache.org/jira/browse/HDFS-11779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
>
> Lists buckets of a given volume. Similar to listVolumes, paging supported via
> prevKey, prefix and maxKeys.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
