[
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=672069&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672069
]
ASF GitHub Bot logged work on HDFS-16283:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Oct/21 17:12
Start Date: 29/Oct/21 17:12
Worklog Time Spent: 10m
Work Description: aihuaxu edited a comment on pull request #3595:
URL: https://github.com/apache/hadoop/pull/3595#issuecomment-954900267
Thanks for the comments. @goiri, @symious and @Hexiaoqiao. We have thought
of the solution completely on the router side to avoid the additional info
exposed in the DFSClient. But the routers need to keep track of the mapping of
the client->files and such mapping needs to be shared across multiple routers.
That brings much complexity and also introduces the high latency on the router
side.
renewLease(String clientName, String path) is a good solution from interface
perspective, but it could cause NameNode overload as @Hexiaoqiao mentioned
since instead of renewing per client, now we are renewing lease per file. This
solution and current implementation will run into scale issue.
Since adding renewLease(String clientName, String nsId) achieves the best
performance, I feel we should still pursue similar idea rather than sacrificing
the performance?
Another thought: maybe we can add an interface of renewLease(String
clientName, List<String> files) to renew lease for all the files of that client
and then the router can resolve to name services and make one call against one
name service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 672069)
Time Spent: 1h 10m (was: 1h)
> RBF: improve renewLease() to call only a specific NameNode rather than make
> fan-out calls
> -----------------------------------------------------------------------------------------
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Reporter: Aihua Xu
> Assignee: Aihua Xu
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes
> are slow, then eventually the router queues are blocked by all renewLease()
> and cause router degradation.
> We will make a change in the client side to keep track of NameNode Id in
> additional to current fileId so routers understand which NameNodes the client
> is renewing lease against.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
