[
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=787495&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-787495
]
ASF GitHub Bot logged work on HDFS-16283:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Jul/22 06:41
Start Date: 04/Jul/22 06:41
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on PR #4524:
URL: https://github.com/apache/hadoop/pull/4524#issuecomment-1173413790
Thanks for quick response.
> Long run client case. There may be many files being written at the same
time.
In my practice, the cost with split-path to renewLease will be under control
even for long running applications, such flink applications (I have not
observed that many files being written concurrently, it will be helpful if any
cases could offer.)
> Multiple destination case. RBF always forwards the renew lease rpc to all
destination name service.
For both create and renewLease (with file path), I think they will apply the
same MountTableResolver for same file. So it does not seem to one issue for
renewLease. Maybe some corner case I do not catch. Please correct me if
something missed.
> the number of renewLease requests between client and rbf will also
increases.
Yes, it is true. I am totally agree. Based on my internal production
cluster, it will be less than 5% increase.
BTW, My consideration here is that it will be more smooth and understandable
only one rbf namespace expose for client, rather than rbf and all namespaces
behind router expose to client. Another side, renewLease is one lightweight
request, less than 5% overhead is acceptable in my opinion.
Of course, the above information is totally based on my internal practice,
maybe some other cases are not included. Very glad to hear more discussions and
suggestions. Thanks.
Issue Time Tracking
-------------------
Worklog Id: (was: 787495)
Time Spent: 4.5h (was: 4h 20m)
> RBF: improve renewLease() to call only a specific NameNode rather than make
> fan-out calls
> -----------------------------------------------------------------------------------------
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Reporter: Aihua Xu
> Assignee: Aihua Xu
> Priority: Major
> Labels: pull-request-available
> Attachments: RBF_ improve renewLease() to call only a specific
> NameNode rather than make fan-out calls.pdf
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes
> are slow, then eventually the router queues are blocked by all renewLease()
> and cause router degradation.
> We will make a change in the client side to keep track of NameNode Id in
> additional to current fileId so routers understand which NameNodes the client
> is renewing lease against.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
