[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786161#comment-16786161
]
CR Hota commented on HDFS-13532:
--------------------------------
[~hexiaoqiao] [~elgoiri] [~brahmareddy]
Sorry for chiming in little late.Thanks for all your thoughts. This is very
good discussion.
In terms of zookeeper as a choice of state store, in the thoughts document I
had uploaded other state stores have also been mentioned. We finally chose
zookeeper as abstractions are already available in current hadoop and it does
allow a push model instead of just a pull model. Through this lookups become
much faster without compromising on consistency. I haven't tested creating
7Millon delegation token znodes, but honestly this is more of a test for
zookeeper itself. BTW, in our clusters majority of jobs complete within 1 day
and so hitting these numbers are highly unlikely anytime in the near future.
Routers not having security feature was a big hindrance in adopting it for any
secure use case irrespective of scale.
However, there is definitely a plan to have a db based state store at some
point. Though with DB based approach lookups would become more expensive and
this would mean other trade offs etc.
"dfs.federation.router.secret.manager.class" does allow plugging-in different
implementations. There is no Jira yet, once we make sure this work in merged
and verified etc, we can focus on creating a new implementation agnostic to zk.
>From migration perspective, both namenodes and routers can co-exist. Routers
>nameservice can be pushed first to whole of yarn (rm and nm) without router
>being default filesystem. When jobs try to access something like
>hdfs://router-nameservice/mydata, rm will use the same filesystem i.e.
>hdfs://router-nameservice to renew tokens. As long as routers are accessed
>using full qualified path names, RM will know where to connect to renew
>tokens. Not sure if I understand this migration issue correctly.
> RBF: Adding security
> --------------------
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Íñigo Goiri
> Assignee: CR Hota
> Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_
> Security delegation token thoughts_updated_3.pdf, Security_for_Router-based
> Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
