[
https://issues.apache.org/jira/browse/HDFS-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ye Ni updated HDFS-15417:
-------------------------
Description:
*Why*
For WebHDFS CREATE, OPEN, APPEND and GETFILECHECKSUM operations, router or
namenode needs to get the datanodes where the block is located, then redirect
the request to one of the datanodes.
However, this chooseDatanode action in router is much slower than namenode,
which directly affects the WebHDFS operations above.
For namenode WebHDFS, it normally takes tens of milliseconds, while router
always takes more than 2 seconds.
*How*
Cache the datanode report in router RPC server. Actively refresh with a
configured interval. Only get the datanode report when necessary in router.
It is a very expense operation where all the time is spent on.
This is only needed when we want to exclude some datanodes or find a random
datanode for CREATE.
was:
*Why*
For WebHDFS CREATE, OPEN, APPEND and GETFILECHECKSUM operations, router or
namenode needs to get the datanodes where the block is located, then redirect
the request to one of the datanodes.
However, this chooseDatanode action in router is much slower than namenode,
which directly affects the WebHDFS operations above.
For namenode WebHDFS, it normally takes tens of milliseconds, while router
always takes more than 2 seconds.
*How*
Only get the datanode report when necessary in router. It is a very expense
operation where all the time is spent on.
This is only needed when we want to exclude some datanodes or find a random
datanode for CREATE.
> RBF: Get the datanode report from cache for federation WebHDFS operations
> -------------------------------------------------------------------------
>
> Key: HDFS-15417
> URL: https://issues.apache.org/jira/browse/HDFS-15417
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: federation, rbf, webhdfs
> Reporter: Ye Ni
> Assignee: Ye Ni
> Priority: Major
>
> *Why*
> For WebHDFS CREATE, OPEN, APPEND and GETFILECHECKSUM operations, router or
> namenode needs to get the datanodes where the block is located, then redirect
> the request to one of the datanodes.
> However, this chooseDatanode action in router is much slower than namenode,
> which directly affects the WebHDFS operations above.
> For namenode WebHDFS, it normally takes tens of milliseconds, while router
> always takes more than 2 seconds.
> *How*
> Cache the datanode report in router RPC server. Actively refresh with a
> configured interval. Only get the datanode report when necessary in router.
> It is a very expense operation where all the time is spent on.
> This is only needed when we want to exclude some datanodes or find a random
> datanode for CREATE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
