[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831823#comment-16831823
]
CR Hota commented on HDFS-13522:
--------------------------------
[~elgoiri] [~ayushtkn] [~csun] Thanks a lot for the discussion and
[~surendrasingh] many thanks for your initial patch. Great to see more interest
in this work.
Based on understanding, below is the problem statement and design exit criteria
we may want to look at. Attached is a file that represents the problem
statement pictorially.
h1. Problem statement
Figure 1 in attached Router+Observer RPC clogging.png shows a typical RPC
mechanism with respect to active namenode and observer namenode. In this case,
observer namenodes are strictly processing read only requests from clients.
Since there is no global lock or contention, RPC queue wait times are lower and
processing times are significantly faster in Observer namenodes when compared
to active namenode. With router based federation, a proxy layer is introduced
that serves incoming client traffic on behalf of the client and performs the
actual action against the downstream namenode. _This server proxy layer
inherits the same server implementation that name nodes utilize_. With a single
RPC queue in router, all read and writes will get intermingled again thus
substantially diminishing the benefits of Observer namenode. Figure 2
illustrates this behavior. This is particularly problematic for rpc latency
sensitive real time engines such as Presto.
h1. Design criteria
On a high level, the design of this feature should help achieve below 2 key
objectives.
# Separation of read vs write queuing in routers to begin with to fundamental
continue maintaining a fast lane access for read calls. This work should create
foundations to help to eventually separate read vs write per nameservice thus
helping solve HDFS-14090.
# Honor existing client configurations around whether or not to access
Observer NN for certain use cases. For ex: a client can currently continue
using ConfiguredFailOverProxy without connecting to Observer. If a client
maintains such preference, routers should honor it and connect to Active
Namenode for read calls as well.
> Support observer node from Router-Based Federation
> --------------------------------------------------
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
> Reporter: Erik Krogen
> Assignee: Chao Sun
> Priority: Major
> Attachments: HDFS-13522.001.patch, Router+Observer RPC clogging.png
>
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
