[
https://issues.apache.org/jira/browse/HDFS-17147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HDFS-17147:
------------------------------
Description:
Suppose we mount table as below:
{code:java}
/dir -> ns0 -> /target/dir
/dir/child1 -> ns0 -> /target/dir/child1
/dir/child2 -> ns0 -> /target/dir/child2
......
/dir/child200 -> ns0 -> /target/dir/child200
{code}
when listing /dir with RBF, it's getting extremely slow as getListing has two
parts:
1. list all children of /target/dir
2. append the rest 200 mount points to the result.
The second part invoke getFileInfo concurrently to make sure mount points are
accessed under rightful permission. But in this case, the first part includes
the result of the second part, and there is no need to append second part
repeatly.
was:
Suppose we mount table as below:
/dir -> ns0 -> /target/dir
/dir/child1 -> ns0 -> /target/dir/child1
/dir/child2 -> ns0 -> /target/dir/child2
......
/dir/child200 -> ns0 -> /target/dir/child200
when listing /dir with RBF, it's getting extremely slow as getListing has two
parts:
1. list all children of /target/dir
2. append the rest 200 mount points to the result.
The second part invoke getFileInfo concurrently to make sure mount points are
accessed under rightful permission. But in this case, the first part includes
the result of the second part, and there is no need to append second part
repeatly.
> RBF: RouterRpcServer getListing become extremely slow when the children of
> the dir are mounted in the same ns.
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17147
> URL: https://issues.apache.org/jira/browse/HDFS-17147
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
> Reporter: Yuanbo Liu
> Priority: Major
>
> Suppose we mount table as below:
>
> {code:java}
> /dir -> ns0 -> /target/dir
> /dir/child1 -> ns0 -> /target/dir/child1
> /dir/child2 -> ns0 -> /target/dir/child2
> ......
> /dir/child200 -> ns0 -> /target/dir/child200
> {code}
>
>
> when listing /dir with RBF, it's getting extremely slow as getListing has two
> parts:
> 1. list all children of /target/dir
> 2. append the rest 200 mount points to the result.
>
> The second part invoke getFileInfo concurrently to make sure mount points are
> accessed under rightful permission. But in this case, the first part includes
> the result of the second part, and there is no need to append second part
> repeatly.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
