[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572181#comment-17572181
]
ZanderXu commented on HDFS-13522:
---------------------------------
[~xkrogen] Thanks for your comment.
bq. I would assume that if your namespaces are so finely segmented that you
have 100+, then each one would be small enough so as to not require the use of
CRS. Are you really running 100+ namespaces, each of which includes Observer
Nodes?
Yes, we have a large HDFS Cluster with 6w+ DN, 100+ namespace, 50+ RBF. And
each namespace contains one active namenode, one standby namenode, and one
observer namenode. Both standby namenode and observer namenode can handle read
requests. Each nameservice has a relatively large QPS, so we enable Observer
Read in all NSs.
bq. I am still in support of a hybrid Design A + B in the interest of
supporting both old and new clients
I'm not against design A. I just think that design B will have more usage
scenarios, so we can accomplish it first.
BTW, can you share me some use cases about Design A if you have?
bq. I am curious about the situation that would lead to the issue of very large
RPC headers as discussed above.
This large RPC is not the large response in Server.class. I just feel that
Design A will caused Client carries a lot of useless NS TxIds to RBF, because
at most scenarios, RBF just proxy request from one client to one downstream NS.
And as the underlying NS increases, client will carries more and more useless
NS TxIds to RBF.
Again, I have nothing against Design A, I just think Design B is more common.
> RBF: Support observer node from Router-Based Federation
> -------------------------------------------------------
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
> Reporter: Erik Krogen
> Assignee: Simbarashe Dzinamarira
> Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, HDFS-13522_proposal_zhengchenyu_v1.pdf, RBF_ Observer
> support.pdf, Router+Observer RPC clogging.png,
> ShortTerm-Routers+Observer.png,
> observer_reads_in_rbf_proposal_simbadzina_v1.pdf,
> observer_reads_in_rbf_proposal_simbadzina_v2.pdf
>
> Time Spent: 20h 10m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
