[
https://issues.apache.org/jira/browse/HDFS-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17436168#comment-17436168
]
Jing Zhao commented on HDFS-16283:
----------------------------------
Checking into the code, looks like besides the router side performance issue,
the whole lease mechanism may need to get updated to support hdfs router.
Currently the DFSClient uses a map(INodeID, DFSOutputStream) to track all the
being-written files. The assumption is that all the being-written files are in
the same nameservice thus there is no INodeID conflict. Now with the support of
router, we may have two files belonging to two different nameservices sharing
the same INodeID (though the possibility is very low in production). So
theoretically we should update the being-written-file map to ((nameservice,
INodeID) – DFSOutputStream).
I understand the concern that with router we do not want client to know
individual nameservices. It would be better if we can still hide nameservices.
We can discuss different solutions in this ticket. In summary I guess we need
to have a new mechanism to align the current INode ID based lease renewal
approach to the new router architecture.
> RBF: improve renewLease() to call only a specific NameNode rather than make
> fan-out calls
> -----------------------------------------------------------------------------------------
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
> Reporter: Aihua Xu
> Assignee: Aihua Xu
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes
> are slow, then eventually the router queues are blocked by all renewLease()
> and cause router degradation.
> We will make a change in the client side to keep track of NameNode Id in
> additional to current fileId so routers understand which NameNodes the client
> is renewing lease against.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
