[
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=786994&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-786994
]
ASF GitHub Bot logged work on HDFS-16283:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Jul/22 08:44
Start Date: 01/Jul/22 08:44
Worklog Time Spent: 10m
Work Description: ZanderXu opened a new pull request, #4524:
URL: https://github.com/apache/hadoop/pull/4524
### Description of PR
[HDFS-16283](https://issues.apache.org/jira/browse/HDFS-16283): RBF: improve
renewLease() to call only a specific NameNode rather than make fan-out calls
Currently RBF will forward the renewLease() rpc to all the available name
services. So the forwarding efficiency will be affected by the unhealthy
downstream name services. And along with as more as NSs are monitored by RBF,
this problem will become more and more serious.
In our prod cluster, there are 70+ nameservices, the phenomenon that
renewLease() rpc is blocked often occurs.
This patch is be used to fix this problem and work well on our cluster, and
the main ideas is:
- Carrying nsId to client when creating a new file
- Store this nsId in DFSOutputStream
- Client renewLease() rpc carries all nsIds of all DFSOutputStream to RBF
- RBF parses the nsIds and forwards the renewLease rpc to the corresponding
name services
Issue Time Tracking
-------------------
Worklog Id: (was: 786994)
Time Spent: 2h 20m (was: 2h 10m)
> RBF: improve renewLease() to call only a specific NameNode rather than make
> fan-out calls
> -----------------------------------------------------------------------------------------
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Reporter: Aihua Xu
> Assignee: Aihua Xu
> Priority: Major
> Labels: pull-request-available
> Attachments: RBF_ improve renewLease() to call only a specific
> NameNode rather than make fan-out calls.pdf
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes
> are slow, then eventually the router queues are blocked by all renewLease()
> and cause router degradation.
> We will make a change in the client side to keep track of NameNode Id in
> additional to current fileId so routers understand which NameNodes the client
> is renewing lease against.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
