[
https://issues.apache.org/jira/browse/HDFS-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178712#comment-16178712
]
Weiwei Yang edited comment on HDFS-12506 at 9/25/17 8:20 AM:
-------------------------------------------------------------
The javadoc warnings seems to be a false alarm, it doesn't understand doc
reference
{code}
/**
* {@link #getRangeKVs(byte[], int, MetadataKeyFilter...)}
**/
{code}
which works perfectly on my intellji IDE. I manually run {{mvn
javadoc:javadoc}}, I got following warnings
{noformat}
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java:45:
warning - Tag @link: reference not found:
org.apache.ratis.statemachine.StateMachine
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java:1274:
warning - Tag @link: can't find newDirectExecutorService() in
org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java:1274:
warning - Tag @link: reference not found: CallerRunsPolicy
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerHandler.java:75:
warning - Tag @link: reference not found:
ChannelHandlerContext#fireExceptionCaught(Throwable)
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocolPB/KeySpaceManagerProtocolServerSideTranslatorPB.java:95:
warning - Tag @link: reference not found:
org.apache.hadoop.ozone.ksm.protocolPB.KeySpaceManagerProtocolPB
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java:76:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java:76:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:164:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:164:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeyManager.java:101:
warning - Tag @link: reference not found: KsmKeyInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:188:
warning - Tag @link: reference not found: KsmKeyInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/netty/RequestContentObjectStoreChannelHandler.java:42:
warning - Tag @link: reference not found: HttpContent
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/netty/RequestDispatchObjectStoreChannelHandler.java:44:
warning - Tag @link: reference not found: HttpRequest
{noformat}
It looks like javadoc plugin doesn't recognize these links. I think we can
commit this, this 2 links are useful to help people know the difference of two
methods. [~linyiqun], any thought on the java doc warnings?
was (Author: cheersyang):
The javadoc warnings seems to be a false alarm, it doesn't understand doc
reference
{code}
/**
* {@link #getRangeKVs(byte[], int, MetadataKeyFilter...)}
**/
{code}
which works perfectly on my intellji IDE. I manually run {{mvn
javadoc:javadoc}}, I got following warnings
{noformat}
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java:45:
warning - Tag @link: reference not found:
org.apache.ratis.statemachine.StateMachine
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java:1274:
warning - Tag @link: can't find newDirectExecutorService() in
org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java:1274:
warning - Tag @link: reference not found: CallerRunsPolicy
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerHandler.java:75:
warning - Tag @link: reference not found:
ChannelHandlerContext#fireExceptionCaught(Throwable)
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocolPB/KeySpaceManagerProtocolServerSideTranslatorPB.java:95:
warning - Tag @link: reference not found:
org.apache.hadoop.ozone.ksm.protocolPB.KeySpaceManagerProtocolPB
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java:76:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java:76:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:164:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:164:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeyManager.java:101:
warning - Tag @link: reference not found: KsmKeyInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:188:
warning - Tag @link: reference not found: KsmKeyInfo
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/netty/RequestContentObjectStoreChannelHandler.java:42:
warning - Tag @link: reference not found: HttpContent
[WARNING]
/Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/netty/RequestDispatchObjectStoreChannelHandler.java:44:
warning - Tag @link: reference not found: HttpRequest
{noformat}
It looks like javadoc plugin doesn't recognize these links. I think we can
commit this, this 2 links are useful to help people know the difference of two
methods.
> Ozone: ListBucket is too slow
> -----------------------------
>
> Key: HDFS-12506
> URL: https://issues.apache.org/jira/browse/HDFS-12506
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
> Reporter: Weiwei Yang
> Assignee: Weiwei Yang
> Priority: Blocker
> Labels: ozoneMerge, performance
> Attachments: HDFS-12506-HDFS-7240.001.patch,
> HDFS-12506-HDFS-7240.002.patch, HDFS-12506-HDFS-7240.003.patch,
> HDFS-12506-HDFS-7240.004.patch, HDFS-12506-HDFS-7240.005.patch,
> HDFS-12506-HDFS-7240.006.patch, HDFS-12506-HDFS-7240.007.patch
>
>
> Generated 3 million keys in ozone, and run {{listBucket}} command to get a
> list of buckets under a volume,
> {code}
> bin/hdfs oz -listBucket http://15oz1.fyre.ibm.com:9864/vol-0-15143 -user wwei
> {code}
> this call spent over *15 seconds* to finish. The problem was caused by the
> inflexible structure of KSM DB. Right now {{ksm.db}} stores keys like
> following
> {code}
> /v1/b1
> /v1/b1/k1
> /v1/b1/k2
> /v1/b1/k3
> /v1/b2
> /v1/b2/k1
> /v1/b2/k2
> /v1/b2/k3
> /v1/b3
> /v1/b4
> {code}
> keys are sorted in nature order so when we do list buckets under a volume e.g
> /v1, we need to seek to /v1 point and start to iterate and filter keys, this
> ends up with scanning all keys under volume /v1. The problem with this design
> is we don't have an efficient approach to locate all buckets without scanning
> the keys.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
