[
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=787485&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-787485
]
ASF GitHub Bot logged work on HDFS-16283:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Jul/22 04:55
Start Date: 04/Jul/22 04:55
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on PR #4524:
URL: https://github.com/apache/hadoop/pull/4524#issuecomment-1173346196
@ZanderXu @ayushtkn, Thanks for your great works here. After a quick glance,
it seems one solution to improve renewLease for RBF.
I would like to share my practice for this issue. I also meet this
renewLease performance issue when upgrade to RBF architecture. When I observed
that renewLease request count to NameNode grew unexpected and time cost
obviously, I try to analysis if it is possible to bring file path as one
parameter to renewLease. After collect `create` and `renewLease` audit, there
are less than 3% renewLease requests to renew different files lease for one
client (generally less than 5 files) in our data warehouse scenario (Maybe it
is not true for other scenarios). Then I try to add a new interface `public
void renewLease(String path, String clientName)` and route request based on
`path` at Router side which is common logic.
Just propose another choice for this improvement. This is not objection for
this PR, just another solution for discussion. If we have reached agreement, I
would like to give deep reviews. Thanks again.
Issue Time Tracking
-------------------
Worklog Id: (was: 787485)
Time Spent: 4h 10m (was: 4h)
> RBF: improve renewLease() to call only a specific NameNode rather than make
> fan-out calls
> -----------------------------------------------------------------------------------------
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Reporter: Aihua Xu
> Assignee: Aihua Xu
> Priority: Major
> Labels: pull-request-available
> Attachments: RBF_ improve renewLease() to call only a specific
> NameNode rather than make fan-out calls.pdf
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes
> are slow, then eventually the router queues are blocked by all renewLease()
> and cause router degradation.
> We will make a change in the client side to keep track of NameNode Id in
> additional to current fileId so routers understand which NameNodes the client
> is renewing lease against.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
