[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17571564#comment-17571564
]
Simbarashe Dzinamarira edited comment on HDFS-13522 at 7/26/22 6:01 PM:
------------------------------------------------------------------------
Thanks [~xuzq_zander] for listing the considerations.
When there are 100+ nameservices then the large RPC header would indeed be a
problem. However, for a few nameservices, I've assumed the larger RPC header is
a better tradeoff that the extra msync call for strong consistency.
*We can do a combination of both Design A and B.*
So Design B (always msync) would be the baseline which doesn't requirement any
client changes. Then if the number of nameservices is under a certain
threshold, the router can send extra information in the header. An old client
ignores this information, which results in Design B behavior, but a new client
transparently forwards this back to the router allowing the router to avoid the
msync (Design A).
My implementation already does follow Design B to support old clients, and a
threshold can be easily added so that Design A is only used in situations where
there aren't too many namespaces.
[~zhengchenyu] can you share how I can access the slack channel. Is there a
slack instance of Apache?
was (Author: simbadzina):
Thanks [~xuzq_zander] for listing the considerations.
When there are 100+ nameservices then the large RPC header would indeed be a
problem. However, for a few nameservices, I've assumed the larger RPC header is
a better tradeoff that the extra msync call for strong consistency.
We can do a combination of both Design A and B.
So Design B (always msync) would be the baseline which doesn't requirement any
client changes. Then if the number of nameservices is under a certain
threshold, the router can send extra information in the header. An old client
ignores this information, which results in Design B behavior, but a new client
transparently forwards this back to the router allowing the router to avoid the
msync (Design A).
My implementation already does follow Design B to support old clients, and a
threshold can be easily added so that Design A is only used in situations where
there aren't too many namespaces.
[~zhengchenyu] can you share how I can access the slack channel. Is there a
slack instance of Apache?
> RBF: Support observer node from Router-Based Federation
> -------------------------------------------------------
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
> Reporter: Erik Krogen
> Assignee: Simbarashe Dzinamarira
> Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, HDFS-13522_proposal_zhengchenyu_v1.pdf, RBF_ Observer
> support.pdf, Router+Observer RPC clogging.png,
> ShortTerm-Routers+Observer.png,
> observer_reads_in_rbf_proposal_simbadzina_v1.pdf,
> observer_reads_in_rbf_proposal_simbadzina_v2.pdf
>
> Time Spent: 20h
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
