[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872119#comment-16872119 ] Brahma Reddy Battula commented on HDFS-13532: - Thanks to all for great discussion. now this is merged to trunk. So going to close this jira. Some pending jira's are moved to HDFS-14603. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832003#comment-16832003 ] CR Hota commented on HDFS-13532: [~hexiaoqiao] [~elgoiri] Sharing some stats on zk testing. We happened to run some more tests on zookeeper token store lately. It was easy to store approximately 2 million tokens. We did not do any tests beyond. While configuring zk, specially for client side bumping up jute.maxbuffer is important. Size should be such that zookeeper client can stream all tokens. This size is dependent on how many tokens are being created and will be streamed. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787509#comment-16787509 ] He Xiaoqiao commented on HDFS-13532: [~elgoiri], [~crh]. Thanks for very helpful suggestions. The migration steps are very clear and helpful! I have to estimate the whole cost to migrate to RBF completely since all our default filesystem is viewfs://nameservice/(include hivemeta, and user applications) and very massive scale, so it will bring high cost to switch hdfs://nameservice. I would like to share information in time and thanks for your help again. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787099#comment-16787099 ] CR Hota commented on HDFS-13532: [~hexiaoqiao] [~elgoiri] I think the confusion is arising as I said earlier is with default FS. In our set-up, defaultFS is still main namenode. Clients(RM or generic clients) trying to access RBF need to specify fully qualified name hdfs://router-nameservice/mydata. Yarn nodes should already have the changes in hdfs-site.xml to expose this new nameservice. Our migration steps are 1. Expose router endpoints to all services first, workflow scheduler, yarn, hive, presto, spark etc WITHOUT making router as the defaultFS. 2. Change ingestion to update hive metastore and point table and partition location to hdfs://router-nameservice instead of hdfs://main-service. 3. Query engines now accessing data that is onboarded on top of router, pass fully qualified hdfs uri let yarn know where to connect to for both read and write (delegation token calls are on a simpler level just write RPC calls such as create, append etc etc). 4. We haven't yet changed defaultFS and will do that much later, probably towards end of q3. We never spend time in investigating how defaultFS can be changed, that would be quite complex honestly as far as i can think. Let's park this discussion till we get more insights from you when you try the actual migration at your end. Share failure logs etc and we can re-visit this. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786430#comment-16786430 ] Íñigo Goiri commented on HDFS-13532: Regarding upgrade, you can do multiple approaches but this is my recommendation. The first step would be to deploy all the Routers with security enabled and delegation tokens. In this moment, you will have both approaches available: * Your regular subclusters directly available through the NameNodes (hdfs://ns0, hdfs://ns1). * The federated namespace accessible through the Routers (hdfs://ns-fed). At this point, independently of YARN settings (NM and RM), you can submit jobs to access data directly (e.g., hdfs://ns0) or through RBF (e.g., hdfs://ns-fed). Here it is important for you to use fully qualified HDFS paths (hdfs://ns-fed/user/user1) and not simple ones (/user/user1). The job will get the DT from the right places. Then, you can have some clients using RBF and others directly going to the Namenode. I think this covers your basic needs. Notice you don't need to do any changes in the NM/RM. After this, you can update the defaultFS for the NM and the RMs separately. Using this approach you don't need to worry about the users as they will go to the right endpoint for the DT. If you want to keep the users using defaultFS instead of fully qualified paths, it will need a fancier rolling upgrade approach. However, if I assume you are switching from viewfs:// to hdfs:// you should be able to do this. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786382#comment-16786382
]
He Xiaoqiao commented on HDFS-13532:
[~crh],[~elgoiri],[~brahmareddy], really appreciate your feedback.
Basically I am concerning 2 points in one word:
* (1) how to gray upgrade HDFS to support RBF with security feature.
* (2) performance cost using ZKDelegationTokenSecretManagerImpl.
And It is clear about (2) performance of ZKDelegationTokenSecretManagerImpl
with my colleague's help. it is OK for me that >5K QPS.
I do not understand about gray upgrade completely. First of all, I would like
to share ideal plan for me to upgrade RBF smoothly: (1) HDFS build on
Federation + ViewFS now. (2) It's better for me to rolling upgrade Client
rather than switch to RBF once time.
[~elgoiri] and [~crh] both mentioned solution with 'Router nameservice' as
following step:
* (1) update YARN(RM/NM) configuration within new router nameservice;
* (2) rolling client to support RBF;
* (3) updete YARN(RM/NM) configuration which include router nameservice config
only;
IIUC, this solution will not solve delegation token issue, since client obtains
DT from router only after step (2) and submit job normally, however executor
will fail when request to NameNode due to DT checks fail, since for some
compute engine (for instance MR) it merges client and NM configuration
together, then executor still request to NameNode directly without proper DT.
To [~crh]
{quote}jobs try to access something like hdfs://router-nameservice/mydata, rm
will use the same filesystem i.e. hdfs://router-nameservice to renew tokens
{quote}
I think it need to enhance compute engine, may be more high-cost.
{quote}Routers not having security feature was a big hindrance in adopting it
for any secure use case irrespective of scale.
{quote}
security feature is also very important for me, I try my best to dig solution
that can transmit to RBF smoothly.
Thanks [~crh], [~elgoiri] again.
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_
> Security delegation token thoughts_updated_3.pdf, Security_for_Router-based
> Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786161#comment-16786161 ] CR Hota commented on HDFS-13532: [~hexiaoqiao] [~elgoiri] [~brahmareddy] Sorry for chiming in little late.Thanks for all your thoughts. This is very good discussion. In terms of zookeeper as a choice of state store, in the thoughts document I had uploaded other state stores have also been mentioned. We finally chose zookeeper as abstractions are already available in current hadoop and it does allow a push model instead of just a pull model. Through this lookups become much faster without compromising on consistency. I haven't tested creating 7Millon delegation token znodes, but honestly this is more of a test for zookeeper itself. BTW, in our clusters majority of jobs complete within 1 day and so hitting these numbers are highly unlikely anytime in the near future. Routers not having security feature was a big hindrance in adopting it for any secure use case irrespective of scale. However, there is definitely a plan to have a db based state store at some point. Though with DB based approach lookups would become more expensive and this would mean other trade offs etc. "dfs.federation.router.secret.manager.class" does allow plugging-in different implementations. There is no Jira yet, once we make sure this work in merged and verified etc, we can focus on creating a new implementation agnostic to zk. >From migration perspective, both namenodes and routers can co-exist. Routers >nameservice can be pushed first to whole of yarn (rm and nm) without router >being default filesystem. When jobs try to access something like >hdfs://router-nameservice/mydata, rm will use the same filesystem i.e. >hdfs://router-nameservice to renew tokens. As long as routers are accessed >using full qualified path names, RM will know where to connect to renew >tokens. Not sure if I understand this migration issue correctly. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784785#comment-16784785
]
Íñigo Goiri commented on HDFS-13532:
{quote}
IIUC, client/jobsubmitter and executors have to switch to RBF in the same time,
otherwise, delegation token check will not pass since they are not matching
distributed from namenode and router.
on another side, majority compute engine run on yarn rely on RM to renew token,
So In on word, it looks that there are no graceful solution to support rolling
upgrade, for instance rolling upgrade client to RBF, then YARN(RM/NM)?
{quote}
As far as I can tell you can have jobs with one defaultFS and NM/RM with a
different one.
You can run jobs against RBF while the NM and the RM use the Namenodes directly.
For moving NM/RM from using the Namenodes to RBF, I'm not sure what the process
would be there.
I think you might be able to transition the RMs first and then the NMs.
I'm not sure if there is an issue here or not.
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_
> Security delegation token thoughts_updated_3.pdf, Security_for_Router-based
> Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784726#comment-16784726
]
He Xiaoqiao commented on HDFS-13532:
Thanks [~brahmareddy] and [~elgoiri] for your detailed comments.
To [~elgoiri],
{quote}If the job is submitted against the Router, then the job can only access
data through RBF.
However, I think this is OK; as I mentioned before you could still have jobs
that query the NameNodes directly.{quote}
IIUC, client/jobsubmitter and executors have to switch to RBF in the same time,
otherwise, delegation token check will not pass since they are not matching
distributed from namenode and router.
on another side, majority compute engine run on yarn rely on RM to renew token,
So In on word, it looks that there are no graceful solution to support rolling
upgrade, for instance rolling upgrade client to RBF, then YARN(RM/NM)?
{quote}For the RM itself, you can transition it from using RBF or not whenever
you want.
{quote}
As mentioned above, I am confused about RM using RBF or not. your more explains
is greatly appreciated.
To [~brahmareddy]
{quote}Did you try it..? do you've failed logs..?{quote}
I am sorry that no time to test this case now, I will offer more info in time
when cover this scenario.
Thanks again.
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_
> Security delegation token thoughts_updated_3.pdf, Security_for_Router-based
> Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784687#comment-16784687
]
Íñigo Goiri commented on HDFS-13532:
{quote}
(1) router takes over delegation tokens management from namenodes at all, (2)
namenode only maintain delegation token request from router. right? IIUC, maybe
there are no graceful gray solution to upgrade clients
{quote}
You can set it up so that clients can still directly work against the Namenodes.
Actually, for you to disable the Namenodes from authenticating clients, you
have to configure it that way and restrict it so only the Routers can get
authenticated.
{quote}
Consider about one job submit to YARN from client which is upgrade to support
RBF, and all delegation tokens are distributed from router, but if yarn still
not upgrade, all executors will authenticate fail to namenode since delegation
token is not matching. Of course this issue is also true if upgrade yarn first
then client.
{quote}
If the job is submitted against the Router, then the job can only access data
through RBF.
However, I think this is OK; as I mentioned before you could still have jobs
that query the Namenodes directly.
Both types of jobs can coexist, it would just be a matter of having a different
{{fs.defaultFS}} for each of them.
For the RM itself, you can transition it from using RBF or not whenever you
want.
Both modes should work at the same time here too.
{quote}
2. any performance test results about zookeeper which manage massive delegation
tokens? I am not very familiar with zookeeper, and if there are obvious
performance differences between zookeeper and memory at namenode before RBF. If
no evaluation, I would like to test it later.
3. if znode number impact performance of delegation token request in zookeeper?
delegation token request ops is very high for a large cluster, for instance,
1000K jobs every day and the maximum lifetime for which a delegation token is
valid set default by 7 days, in the worst case, it will backlog 7000K znodes at
all. some risk for more large cluster?
{quote}
The scale we have used internally with ZK is not large enough for me to give
you a proper answer here.
I think [~crh] had some larger tests.
However, the State Store is pluggable so one could use other approaches like:
HDFS-13245 (still under review) or HDFS-10630.
The secret manager is only in ZK ({{ZKDelegationTokenSecretManagerImpl}} based
on {{ZKDelegationTokenSecretManager}}) but should be easy to extend.
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_
> Security delegation token thoughts_updated_3.pdf, Security_for_Router-based
> Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784661#comment-16784661
]
Brahma Reddy Battula commented on HDFS-13532:
-
{quote}(1) router takes over delegation tokens management from namenodes at
all, (2) namenode only maintain delegation token request from router. right?
IIUC, maybe there are no graceful gray solution to upgrade clients
{quote}
Yes, DelegationToken Management is moved to Router.
{quote} Consider about one job submit to YARN from client which is upgrade to
support RBF, and all delegation tokens are distributed from router, but if yarn
still not upgrade, all executors will authenticate fail to namenode since
delegation token is not matching. Of course this issue is also true if upgrade
yarn first then client.
{quote}
Did you try it..? do you've failed logs..? As there is no client side changes
for this, should not be problem and jobconf.xml will be passed to
ResourceManager so RM also will connect to router to validate the tokens. Need
to check JHS/ATS when mount point and configured history location is different.
{quote}2. any performance test results about zookeeper which manage massive
delegation tokens? I am not very familiar with zookeeper, and if there are
obvious performance differences between zookeeper and memory at namenode before
RBF. If no evaluation, I would like to test it later.
3. if znode number impact performance of delegation token request in zookeeper?
delegation token request ops is very high for a large cluster, for instance,
1000K jobs every day and the maximum lifetime for which a delegation token is
valid set default by 7 days, in the worst case, it will backlog 7000K znodes at
all. some risk for more large cluster?
{quote}
May be [~elgoiri] or [~crh] can update on this.
{quote}4. any plan to support different approach and let user to choice?
{quote}
Yes, it's pluggable. can configure using
"dfs.federation.router.secret.manager.class".
[~crh] and [~elgoiri] do correct me if I am wrong.
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_
> Security delegation token thoughts_updated_3.pdf, Security_for_Router-based
> Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784327#comment-16784327 ] He Xiaoqiao commented on HDFS-13532: Thanks for the great works here, I have followed RBF recently and sorry for no-timely questions. I found that branch-HDFS-13891 implement Approach#1, and it works well in my test env. And some confusion about Approach #1: 1. any suggestions or guide for upgrade gracefully? Approach #1 based on two point: (1) router takes over delegation tokens management from namenodes at all, (2) namenode only maintain delegation token request from router. right? IIUC, maybe there are no graceful gray solution to upgrade clients. Consider about one job submit to YARN from client which is upgrade to support RBF, and all delegation tokens are distributed from router, but if yarn still not upgrade, all executors will authenticate fail to namenode since delegation token is not matching. Of course this issue is also true if upgrade yarn first then client. 2. any performance test results about zookeeper which manage massive delegation tokens? I am not very familiar with zookeeper, and if there are obvious performance differences between zookeeper and memory at namenode before RBF. If no evaluation, I would like to test it later. 3. if znode number impact performance of delegation token request in zookeeper? delegation token request ops is very high for a large cluster, for instance, 1000K jobs every day and the maximum lifetime for which a delegation token is valid set default by 7 days, in the worst case, it will backlog 7000K znodes at all. some risk for more large cluster? 4. any plan to support different approach and let user to choice? Please correct me if there are something wrong. Thanks again. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677551#comment-16677551 ] Brahma Reddy Battula commented on HDFS-13532: - Thanks All.I just committed kerboes patch (HDFS-12284) to HDFS-13891 branch. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677278#comment-16677278 ] CR Hota commented on HDFS-13532: +1 on [~brahmareddy] idea. Its is a good idea to merge these. Since we don't have changes to core, its much easier to manage changes and work on a single branch specially for folks who are putting patches for security and overall stabilization. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677261#comment-16677261
]
Anu Engineer commented on HDFS-13532:
-
{quote}HDFS-13532 and HDFS-13891 are intended to be RBF only and not touch the
core at all.
{quote}
Cool, if you think they have no extra payload and will not hurt during the
merge time, please feel free. Just be cognizant that we don't put other
features in.
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_
> Security delegation token thoughts_updated_3.pdf, Security_for_Router-based
> Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677215#comment-16677215 ] Brahma Reddy Battula commented on HDFS-13532: - AFAIK, we are not targeting any new feature apart from the security in RBF.HDFS-13891 is for stabilisation(HDFS-13891) where we can include this security feature also. and Yes,Core changes will be handled separately as they might not from RBF module. [~elgoiri] Please correct me if I am wrong. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677197#comment-16677197
]
Íñigo Goiri commented on HDFS-13532:
HDFS-13532 and HDFS-13891 are intended to be RBF only and not touch the core at
all.
In the moment that any of the JIRAs requires changes to the core, they should
not be in this branch but somewhere else.
Anyway, we can verify this by doing a diff between the branches and trunk
before voting.
At this point this is true:
{code}
M hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml
M
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java
M
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java
M
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java
M
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterHttpServer.java
M
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
M
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
M
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml
M
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/RouterHDFSContract.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/SecurityConfUtil.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractAppendSecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractConcatSecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractCreateSecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractDeleteSecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractGetFileStatusSecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractMkdirSecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractOpenSecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractRenameSecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractRootDirectorySecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractSeekSecure.java
A
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/fs/contract/router/TestRouterHDFSContractSetTimesSecure.java
M
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/MiniRouterDFSCluster.java
{code}
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_
> Security delegation token thoughts_updated_3.pdf, Security_for_Router-based
> Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677192#comment-16677192
]
Anu Engineer commented on HDFS-13532:
-
{quote}It's better to have one branch for one feature.
{quote}
Agree, but security is one feature for RBF. There are other features also, the
earlier issue was the everything was combined under a single JIRA. The RBF
security work should have very minimal changes to HDFS codebase and should be
easy to call a vote for. Unfortunately some of the other proposals are too
complex and impacts the core HDFS. So it might be better to have different
branches, otherwise the merge will get stuck on some feature that is not
related to RBF security.
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_
> Security delegation token thoughts_updated_3.pdf, Security_for_Router-based
> Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677185#comment-16677185 ] Íñigo Goiri commented on HDFS-13532: I agree, maintaining 2 branches and merging both (with a vote and everything) will add too much overhead. I think having a single branch (HDFS-13891) for the merges would be better in my opinion. We need to make HDFS-13532 to target HDFS-13891 though. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677177#comment-16677177 ] Brahma Reddy Battula commented on HDFS-13532: - Hi All, It's better to have one branch for one feature.Unfortunately there are two branches(HDFS-13891 and HDFS-13532) in RBF now. Maintaince will be costly and voting also need to do twice..Hence I am proposing better to have one branch. AS there are some commits went into HDFS-13891,I feel,all security ( which target to HDFS-13532) can be committed to HDFS -13891. any thoughts..? > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647151#comment-16647151 ] CR Hota commented on HDFS-13532: [~elgoiri] [~anu] Thanks for your good suggestions. I am going to keep security work agnostic to the audit logging for now. Lets revisit the auditing piece once we have the security functionality ready. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647035#comment-16647035 ] Íñigo Goiri commented on HDFS-13532: I think it is fine to start without the audit logging; the basic proxy user we talked about is a good starting point. Once we have the basis we can see if we move HDFS-13293 to this branch as part of security or not. For now, let's leave it as is and let's focus on adding the basic kerberos integration (HDFS-12284) including DT (HDFS-13358). > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647034#comment-16647034 ] Anu Engineer commented on HDFS-13532: - In ozone we have a generic pluggable Audit interface and a parser for that. We use that for Auditing OzoneManager, SCM and Datanodes. Would you like us to bring that over, it will make the code development easier and will come with its own parser too. cc:[~dchitlangia] who wrote that. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647026#comment-16647026 ] CR Hota commented on HDFS-13532: [~elgoiri] [~brahmareddy] [~ajayydv] [~zhengxg3] For this 1st phase of security, am mainly focusing on the functionality and omitting out audit logging (HDFS-13293 and HDFS-13248), since we are yet to merge the audit logging patches and converge on ideas there. Once audit logging for router in general irrespective of security is ironed out (I plan to spend sometime on that area as well), we can add audit logging into the security features in a separate jira. What do you folks think ? > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636060#comment-16636060 ] CR Hota commented on HDFS-13532: All, I was able to code a small prototype based on earlier feedback on the designs. Have set-up a meeting for everyone to join and share thoughts on the prototype and design. Time - Oct 8th 2018, 3-4 PM PST This is the zoom link, [https://uber.zoom.us/j/273426631] > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623943#comment-16623943 ] CR Hota commented on HDFS-13532: [~brahmareddy] Updated the document with your comments. Yes Cons mentioned earlier in Approach 1 is invalid. Irrespective of using tokens for auth, even in kerberos route, a client side service ticket is cached and when a service ticket is presented to namenode, the namenode does NOT connect to KDC for verification. So increased load on KDC isn't a valid point as mentioned in my first document. Am working on submitting the design based on Approach 1 and meanwhile also looking at creating a quick prototype that can demonstrate approach 1's feasibility. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, RBF _ Security delegation > token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf, RBF_ > Security delegation token thoughts_updated_3.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617749#comment-16617749
]
Brahma Reddy Battula commented on HDFS-13532:
-
[~crh] thanks for updating.
As discussed in call, Following Cons for approach 1 are still valid, as Router
also token(act as proxy user) so auth can be done through token.
{quote}bq. Without delegation token use namenodes will end up putting all the
load on KDC for kerberos ticket verification. This will defeat one of the main
rationales behind why delegation tokens were introduced in namenode.
bq. Performance of namenodes will deteriorate further as network calls need to
be made to kdc for ticket verification instead of in memory cache of delegation
tokens that is maintained currently.
{quote}
and once after updating in statestore then we can return ack to the client.
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf, RBF _
> Security delegation token thoughts_updated.pdf, RBF _ Security delegation
> token thoughts_updated_2.pdf, RBF-DelegationToken-Approach1b.pdf,
> Security_for_Router-based Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612667#comment-16612667 ] CR Hota commented on HDFS-13532: Thanks everyone for all the reviews so far. MoM # Everyone tilted towards Approach 1 based on the pros and cons outlined. # Anu raised a valid point about multi domain cluster set-ups. He would like us to update the document with thoughts on this area. # Inigo felt we should start prototyping approach 1. # Brahma also felt Approach 1 would be better. # Everyone more or less agreed that Approach 1 is also easy to implement. Attaching an updated document which contains some initial information around multi domain. [~anu] could you please to add more context around it. As of now, secured router could facade multiple hdfs clusters that all work on the same domain. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609461#comment-16609461 ] Ajay Kumar commented on HDFS-13532: --- Agree with [~brahmareddy]. Approach 1 looks more clean implementation. (maintaining a composite token as in approach 2 comes with its own issues.) > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608515#comment-16608515 ] Brahma Reddy Battula commented on HDFS-13532: - [~crh] thanks for organising the meeting and detailed design doc.Hope you can update MOM. I am favourable to approach 1(which complete moves token life cycle to Router).As, we need to consider the additional cost also. 1) Please update cons for approach 1. 2) For syncing token across the routers,May be we can refresh/sync thread like HDFS-13443 3) Might need to handle KMS token also.? [~daryn]/[~lmccay]/[~vinayrpet] if you get chance,kindly review the design. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606388#comment-16606388 ] CR Hota commented on HDFS-13532: Updated the initial document with a easy to read comparison table and few points around common cases for both approaches and specific to each approach. Lets discuss on these points in the meeting tomorrow. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, RBF _ > Security delegation token thoughts_updated.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601731#comment-16601731 ] Surendra Singh Lilhore commented on HDFS-13532: --- Thanks [~crh] for scheduling meeting. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599449#comment-16599449 ] CR Hota commented on HDFS-13532: All, Have set-up a meeting for everyone to join and discuss the design. Time - Sep7th 2018, 2-3 PM PST This is the zoom link, [https://uber.zoom.us/j/372628408] > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573454#comment-16573454 ] Ajay Kumar commented on HDFS-13532: --- [~crh], sure, i work out of PST. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572212#comment-16572212 ] CR Hota commented on HDFS-13532: [~ajayydv] [~elgoiri] Suggest we have a call to discuss the approaches and converge. Thoughts? > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566108#comment-16566108 ] Ajay Kumar commented on HDFS-13532: --- [~crh], i had a offline discussion with [~jnp], [~xyao] and [~arpitagarwal] on this. If ServiceTicket is cached, Router will not be hammering KDC for each request. If other security experts in community believe that this will be an issue may be we can tweak our approach 1 to mitigate this issue. Attached [^RBF _ Security delegation token thoughts.pdf] to discuss an slightly modified approach. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, > RBF-DelegationToken-Approach1b.pdf, Security_for_Router-based > Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565634#comment-16565634 ] CR Hota commented on HDFS-13532: Thanks [~ajayydv] for going through the doc. The main discussion (around cons for Approach 1) was around avoiding calls to KDC. Router does maintain a pool of connections, but that pool/connection gets recycled every x interval and new connections are created if needed again. The lesser this architecture overall relies on KDC, the better router can perform as a pure proxy with lower latencies. With end to end delegation token route, router remains more aligned as a proxy rather than a gateway. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, > Security_for_Router-based Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[
https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564529#comment-16564529
]
Ajay Kumar commented on HDFS-13532:
---
[~crh], thanks for uploading the document.
{quote}Without delegation token use namenodes will end up putting all the load
on KDC for
kerberos ticket verification. This will defeat one of the main rationales
behind why
delegation tokens were introduced in namenode.
● Performance of namenodes will deteriorate further as network calls need to be
made to
kdc for ticket verification instead of in memory cache of delegation tokens
that is
maintained currently.{quote}
Could you please share more details on discussions around cons mentioned for
Approach 1. AFAIK kerberos auth is expensive operation but once connection is
established for router our rpc connections will be pooled.
> RBF: Adding security
>
>
> Key: HDFS-13532
> URL: https://issues.apache.org/jira/browse/HDFS-13532
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Íñigo Goiri
>Assignee: Sherwood Zheng
>Priority: Major
> Attachments: RBF _ Security delegation token thoughts.pdf,
> Security_for_Router-based Federation_design_doc.pdf
>
>
> HDFS Router based federation should support security. This includes
> authentication and delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549714#comment-16549714 ] CR Hota commented on HDFS-13532: [~elgoiri] [~xiaochen] The assumption is that renewals will also go through the router, router issuing DTs and clients bypassing router and directly talking to namenodes for renewal/cancellation anyways won't work. As far as renewals are concerned, from clients perspective it will be one single call, which will either succeed or fail based on how renewals went through for all downstream name nodes. These details will be covered in the design document of router delegation token in [HDFS-13358|https://issues.apache.org/jira/browse/HDFS-13358]. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, > Security_for_Router-based Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545863#comment-16545863 ] Íñigo Goiri commented on HDFS-13532: Thanks [~crh] for [^RBF _ Security delegation token thoughts.pdf]. Approach #1 has the flavor of a gateway and I think that Approach #2 is the closest to the HDFS philosophy. I would vote for that #2 which is close to what [~zhengxg3] proposed in [^Security_for_Router-based Federation_design_doc.pdf]. Regarding approach #2: * As you mentioned, it imposes some restrictions on the State Store but we can implement the fully consistent implementation in paralle. * Merging and storing the tokens can be somewhat similar to what the NN currently does. It'd be good to have more details but we could leave those details to HDFS-13358. Anyway, once we agree that #2 is the right way to do, we can do a detailed document. Regarding the questions from [~xiaochen], I think that the assumption is correct: the user would authenticate to the Router using regular Kerberos and that's a strong requirement for the Router to forward to the Routers. Regarding the token renewal, the Router would be the one forwarding the RM renewal/cancellation. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, > Security_for_Router-based Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545842#comment-16545842 ] CR Hota commented on HDFS-13532: [~elgoiri] [~zhengxg3] [~xiaochen] The document I uploaded now, holds some thoughts on delegation tokens in Router. Take a look and see if anything is missing in terms high level approach. Am currently working on detailed design document. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: RBF _ Security delegation token thoughts.pdf, > Security_for_Router-based Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517575#comment-16517575 ] Xiao Chen commented on HDFS-13532: -- Thanks for the work here [~zhengxg3] and all. The last page of the doc looks familiar. :) Some high level questions from the doc. I have not followed RBF closely and my apologies if these are stupid questions... * I second what Inigo said above. It's not clear to me how DTr is used. * It looks like we'll add the same mechanism to the router, so clients can auth with kerberos, then get a delegation token for subsequent authentications. Is this understanding correct? * I'm not a very security person - the router proxying as client part seems fine. But IMO that should only work if the client auth'ed via kerberos; if client->router auth is dt, then router should not auth to NN via kerberos, but only via the provided DTnn. * Who's gonna renew the router tokens? Tokens from different NNs may have different expiration time, hence need to be renewed at different intervals. RM currently does this, it's kinda nice to reuse RM to handle the DTr token renewal / cancelation. * [~daryn] at one point mentioned he's working on some token issuer interface. Not sure if it will benefit/collide with the work here. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: Security_for_Router-based Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516050#comment-16516050 ] Íñigo Goiri commented on HDFS-13532: Thanks [~zhengxg3] for the document. Could you add more details on how the Router does the mapping between the DT from the Router (e.g., step 8) to the federated token? >From what we saw in the prototypes, we were getting a DT with a different >signature and we had to do guesses to map it to the DT stored. Can you give more details on how the federated token is created and managed? For example, how do we propagate the federated tokens across Routers, etc. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: Security_for_Router-based Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515966#comment-16515966 ] Sherwood Zheng commented on HDFS-13532: --- Here I upload the draft of the design doc. Please review and let me know if there is anything I am missing. Will redraw the graph. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > Attachments: Security_for_Router-based Federation_design_doc.pdf > > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491070#comment-16491070 ] Sherwood Zheng commented on HDFS-13532: --- [~arpitagarwal] Thank you very much. I am currently stuck on an issue for a long time, I've been investigating it, asking around and trying multiple ways to solve it, but didn't quite get it right. Maybe it's better to post my current working version to the security branch and get advice from those Hadoop security. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475154#comment-16475154 ] Arpit Agarwal commented on HDFS-13532: -- [~zhengxg3], please don't hesitate to ask for help if you have any questions or get stuck on the design. There are many experts on Hadoop security (I'm not one of them :)) who can provide guidance. Another option is to have a community webex to discuss any open questions around RBF+Kerberos. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475123#comment-16475123 ] Anu Engineer commented on HDFS-13532: - Done, please pull when you get a chance. aengineer ~/a/hadoop> git push origin HDFS-13532 Total 0 (delta 0), reused 0 (delta 0) To https://git-wip-us.apache.org/repos/asf/hadoop.git * [new branch] HDFS-13532 -> HDFS-13532 > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475120#comment-16475120 ] Anu Engineer commented on HDFS-13532: - I will create one right now. --Anu > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475097#comment-16475097 ] Íñigo Goiri commented on HDFS-13532: [~anu], do you mind creating the branch? > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474746#comment-16474746 ] Anu Engineer commented on HDFS-13532: - Let us also do this work in a branch, please. > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13532) RBF: Adding security
[ https://issues.apache.org/jira/browse/HDFS-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466791#comment-16466791 ] Íñigo Goiri commented on HDFS-13532: [~zhengxg3] do you mind adding a design doc here for this as [~daryn] asked in HDFS-12284? > RBF: Adding security > > > Key: HDFS-13532 > URL: https://issues.apache.org/jira/browse/HDFS-13532 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Sherwood Zheng >Priority: Major > > HDFS Router based federation should support security. This includes > authentication and delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
