[
https://issues.apache.org/jira/browse/HDFS-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279190#comment-17279190
]
Fengnan Li commented on HDFS-15757:
-----------------------------------
[~hexiaoqiao] Thanks for the question. There are three latencies:
1. rpc queue time, this is measured the time in the RPC queue, which is not
related with the change.
2. rpc processing time, this is measured before the actual proxy op (get tcp
connection and talk to nn), which is not related.
3. proxy time, this is directly impacted since the change improves
getConnection() a lot. I have done some flamegraphes for Router to understand
the performance bottleneck and often I can see getConnection() in the stack
taking a lot of time. With this change, connections are actually maintained as
Active as possible. v.s. previously the connection left not quite closed and
hitting the connection cap for the pool thus no more active connection can be
created.
>From the last graph I included in the doc we can see the ProxyTime is improved.
Feel free to give a try in your setup. It's always good to have a second eye on
it.
> RBF: Improving Router Connection Management
> -------------------------------------------
>
> Key: HDFS-15757
> URL: https://issues.apache.org/jira/browse/HDFS-15757
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
> Reporter: Fengnan Li
> Assignee: Fengnan Li
> Priority: Major
> Labels: pull-request-available
> Attachments: RBF_ Improving Router Connection Management_v2.pdf, RBF_
> Improving Router Connection Management_v3.pdf, RBF_ Router Connection
> Management.pdf
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We have seen high number of connections from Router to namenodes, leaving
> namenodes unstable.
> This ticket is trying to reduce connections through some changes. Please take
> a look at the design and leave comments.
> Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
