[
https://issues.apache.org/jira/browse/HDFS-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17152369#comment-17152369
]
Hudson commented on HDFS-15417:
-------------------------------
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18413 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18413/])
HDFS-15417. RBF: Get the datanode report from cache for federation (github: rev
e820baa6e6f7e850ba62cbf150d760bd0ea6d0e0)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterWebHdfsMethods.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java
> RBF: Get the datanode report from cache for federation WebHDFS operations
> -------------------------------------------------------------------------
>
> Key: HDFS-15417
> URL: https://issues.apache.org/jira/browse/HDFS-15417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: federation, rbf, webhdfs
> Reporter: Ye Ni
> Assignee: Ye Ni
> Priority: Major
>
> *Why*
> For WebHDFS CREATE, OPEN, APPEND and GETFILECHECKSUM operations, router or
> namenode needs to get the datanodes where the block is located, then redirect
> the request to one of the datanodes.
> However, this chooseDatanode action in router is much slower than namenode,
> which directly affects the WebHDFS operations above.
> For namenode WebHDFS, it normally takes tens of milliseconds, while router
> always takes more than 2 seconds.
> *How*
> Cache the datanode report in router RPC server. Actively refresh with a
> configured interval. Only get the datanode report when necessary in router.
> It is a very expense operation where all the time is spent on.
> This is only needed when we want to exclude some datanodes or find a random
> datanode for CREATE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
