[
https://issues.apache.org/jira/browse/HDFS-13274?focusedWorklogId=787727&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-787727
]
ASF GitHub Bot logged work on HDFS-13274:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 05/Jul/22 04:20
Start Date: 05/Jul/22 04:20
Worklog Time Spent: 10m
Work Description: ZanderXu opened a new pull request, #4531:
URL: https://github.com/apache/hadoop/pull/4531
### Description of PR
Jira: [HDFS-13274](https://issues.apache.org/jira/browse/HDFS-13274)
[HADOOP-13144](https://issues.apache.org/jira/browse/HADOOP-13144)
introduces the ability to create multiple connections for the same user and use
different sockets. The RouterRpcClient should use this approach to get a better
throughput.
In my practice, I configured `dfs.federation.router.connection.pool-size=6`
and found some abnormal cases:
- RBF frequently to create socket with NN during forwarding request.
- There are a large number of connection threads with name node. (There are
70+ downstream name services)
After tracing the code and found that we can optimize the mechanism of
round-robin obtaining connection.
- One `ConnectionContext` can support handler some concurrent requests at
the same time, such as 5
- We can mark one `ConnectionContext` usable when the `numThreads` is less
than `maxConcurrencyPerConn`
- About `getConnection`, we can returns the previous usable connection in
order
- If there no usable connections, we can return one connection round-robin
After using the above solution, we effectively controlled the number of
connections between RBF and NameNode, and there is almost no the phenomenon of
creating sockets during forwarding requests.
Issue Time Tracking
-------------------
Worklog Id: (was: 787727)
Remaining Estimate: 0h
Time Spent: 10m
> RBF: Extend RouterRpcClient to use multiple sockets
> ---------------------------------------------------
>
> Key: HDFS-13274
> URL: https://issues.apache.org/jira/browse/HDFS-13274
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Íñigo Goiri
> Assignee: Íñigo Goiri
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> HADOOP-13144 introduces the ability to create multiple connections for the
> same user and use different sockets. The RouterRpcClient should use this
> approach to get a better throughput.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
