[jira] [Comment Edited] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[ https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931888#comment-16931888 ] Íñigo Goiri edited comment on HDFS-14609 at 9/17/19 10:54 PM: -- +1 on [^HDFS-14609.006.patch]. I think [~crh] already gave his official blessing. [~tasanuma] and [~eyang] seem supportive but let's get an official +1 to make sure we are all in the same page. (I cannot wait for this failing unit test to go away, the flaky part is still there but it's not as annoying and HDFS-14461 should take care of it). was (Author: elgoiri): +1 on [^HDFS-14609.006.patch]. I think [~crh] already gave his official blessing. [~tasanuma] and [~eyang] seems supportive but let's get an official +1 to make sure we are all in the same page. (I cannot wait for this failing unit test to go away, the flaky part is still there but it's not as annoying and HDFS-14461 should take care of it). > RBF: Security should use common AuthenticationFilter > > > Key: HDFS-14609 > URL: https://issues.apache.org/jira/browse/HDFS-14609 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: CR Hota >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14609.001.patch, HDFS-14609.002.patch, > HDFS-14609.003.patch, HDFS-14609.004.patch, HDFS-14609.005.patch, > HDFS-14609.006.patch > > > We worked on router based federation security as part of HDFS-13532. We kept > it compatible with the way namenode works. However with HADOOP-16314 and > HDFS-16354 in trunk, auth filters seems to have been changed causing tests to > fail. > Changes are needed appropriately in RBF, mainly fixing broken tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[
https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927286#comment-16927286
]
Chen Zhang edited comment on HDFS-14609 at 9/11/19 6:47 AM:
Hi [~crh], thanks for your response, I've some explaination in this [comment
|https://issues.apache.org/jira/browse/HDFS-14609?focusedCommentId=16907486=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16907486]about
the change of test's behavior, FYI.
For your question, the answer is :
{quote} # For case {{testGetDelegationToken()}}, the server address is set by
WebHdfsFileSystem after it get the response, the original address is the
address of RouterRpcServer. Since we now send request by http connection
directly, it's unnecessary to reset the address, so I removed this assert
# For the case {{testCancelDelegationToken()}}, the {{InvalidToken}} exception
is also generated by WebHdfsFileSystem and the logic is very complex, I think
it's also unnecessary to keep this assert, I use the 403 detection
instead.{quote}
was (Author: zhangchen):
Hi [~crh], thanks for your response, I've some explaination in this comment
about the change of test's behavior, FYI.
For your question, the answer is :
{quote} # For case {{testGetDelegationToken()}}, the server address is set by
WebHdfsFileSystem after it get the response, the original address is the
address of RouterRpcServer. Since we now send request by http connection
directly, it's unnecessary to reset the address, so I removed this assert
# For the case {{testCancelDelegationToken()}}, the {{InvalidToken}} exception
is also generated by WebHdfsFileSystem and the logic is very complex, I think
it's also unnecessary to keep this assert, I use the 403 detection
instead.{quote}
> RBF: Security should use common AuthenticationFilter
>
>
> Key: HDFS-14609
> URL: https://issues.apache.org/jira/browse/HDFS-14609
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: CR Hota
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14609.001.patch, HDFS-14609.002.patch,
> HDFS-14609.003.patch, HDFS-14609.004.patch, HDFS-14609.005.patch
>
>
> We worked on router based federation security as part of HDFS-13532. We kept
> it compatible with the way namenode works. However with HADOOP-16314 and
> HDFS-16354 in trunk, auth filters seems to have been changed causing tests to
> fail.
> Changes are needed appropriately in RBF, mainly fixing broken tests.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[
https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924776#comment-16924776
]
Chen Zhang edited comment on HDFS-14609 at 9/7/19 8:42 AM:
---
Thanks [~elgoiri] for your review and comments.
{quote}TestRouterWithSecureStartup#HTTP_KERBEROS_PRINCIPAL_CONF_KEY, don't we
have this defined somewhere else?{quote}
The conf key is joined by the {{HttpServer2#authFilterConfigurationPrefix}} and
".type" string, so I thought that there might not be a complete key already
defined in somewhere else. But I'm wrong,
{{CommonConfigurationKeysPublic#HADOOP_HTTP_AUTHENTICATION_TYPE}} defines it.
I'll update the patch to use it instead.
{quote}BTW, I've seen the use of equivalents to NoAuthFilterInitializer, don't
we have this already in the common test lib?{quote}Can you point out where is
the equivalents of NoAuthFilterInitializer? I try to find it by going through
all the sub class of {{FilterInitializer}}, but not found.
was (Author: zhangchen):
Thanks [~elgoiri] for your review and comments.
{quote}TestRouterWithSecureStartup#HTTP_KERBEROS_PRINCIPAL_CONF_KEY, don't we
have this defined somewhere else?{quote}
The conf key is joined by the {{HttpServer2#authFilterConfigurationPrefix}} and
".type" string, so I thought that there might not be a complete key already
defined in somewhere else. But I'm wrong,
{{CommonConfigurationKeysPublic#HADOOP_HTTP_AUTHENTICATION_TYPE}} defines it.
I'll update the patch to use this instead.
{quote}BTW, I've seen the use of equivalents to NoAuthFilterInitializer, don't
we have this already in the common test lib?{quote}Can you point out where is
the equivalents of NoAuthFilterInitializer? I try to find it by going through
all the sub class of {{FilterInitializer}}, but not found.
> RBF: Security should use common AuthenticationFilter
>
>
> Key: HDFS-14609
> URL: https://issues.apache.org/jira/browse/HDFS-14609
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: CR Hota
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14609.001.patch, HDFS-14609.002.patch,
> HDFS-14609.003.patch, HDFS-14609.004.patch
>
>
> We worked on router based federation security as part of HDFS-13532. We kept
> it compatible with the way namenode works. However with HADOOP-16314 and
> HDFS-16354 in trunk, auth filters seems to have been changed causing tests to
> fail.
> Changes are needed appropriately in RBF, mainly fixing broken tests.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[ https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916806#comment-16916806 ] Chen Zhang edited comment on HDFS-14609 at 8/27/19 3:30 PM: Thanks [~crh] for your response, created another Jira HDFS-14784 to track the changes on hadoop-hdfs. Also cc [~tasanuma], could you help to take look if you have time? Thanks. was (Author: zhangchen): Thanks [~crh] for your response, created another Jira HDFS-14784 to track the changes on hadoop-hdfs. Also cc [~tasanuma], please help to take look if you have time, thanks. > RBF: Security should use common AuthenticationFilter > > > Key: HDFS-14609 > URL: https://issues.apache.org/jira/browse/HDFS-14609 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: CR Hota >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14609.001.patch, HDFS-14609.002.patch > > > We worked on router based federation security as part of HDFS-13532. We kept > it compatible with the way namenode works. However with HADOOP-16314 and > HDFS-16354 in trunk, auth filters seems to have been changed causing tests to > fail. > Changes are needed appropriately in RBF, mainly fixing broken tests. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[
https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907486#comment-16907486
]
Chen Zhang edited comment on HDFS-14609 at 8/14/19 5:47 PM:
Thanks [~tasanuma] for providing the old revision of HDFS-13891, it's very
helpful.
I've fixed these 2 tests, here is some detail;
h3. TestRouterWithSecureStartup#testStartupWithoutSpnegoPrincipal
HADOOP-16314 and HADOOP-16354 made some changes which breaks the test:
# Added an AuthFilterInitializer, which using
{{hadoop.http.authentication.kerberos.principal}} ** instead of
{{dfs.web.authentication.kerberos}}{{.principal}} to initialize kerberos
# {{hadoop.http.authentication.kerberos.principal}} has a default value, so
even we don't configure this key, the cluster will still start normally
h3. TestRouterHttpDelegationToken
# HDFS-14434 ignores user.name query parameter in secure WebHDFS, and the
initial version of this test leveraged this parameter to bypass the kerberos
authentication, so after HDFS-14434, it's not work. I added a set of methods to
send request by http connection instead of {{WebHdfsFileSystem}} to make it
continue working.
# HADOOP-16314 changed configuration-key of the authentication filter from
{{dfs.web.authentication.filter}} to {{hadoop.http.filter.initializers}}, so I
added an {{NoAuthFilterInitializer}} to initialize {{NoAuthFilter}}
# For case {{testGetDelegationToken()}}, the server address is set by
WebHdfsFileSystem after it get the response, the original address is the
address of RouterRpcServer. Since we now send request by http connection
directly, it's unnecessary to reset the address, so I removed this assert
# For the case {{testCancelDelegationToken()}}, the {{InvalidToken}} exception
is also generated by WebHdfsFileSystem and the logic is very complex, I think
it's also unnecessary to keep this assert, so I using the 403 detection instead.
In the trunk code, the config {{dfs.web.authentication.filter}} is not used
anywhere, I propose to deprecate this config, I'll track this in another Jira.
was (Author: zhangchen):
Thanks [~tasanuma] for providing the old revision of HDFS-13891, it's very
helpful.
I've fixed these 2 tests, here is some detail;
h3. TestRouterWithSecureStartup#testStartupWithoutSpnegoPrincipal
HADOOP-16314 and HADOOP-16354 made some changes which breaks the test:
# Added an AuthFilterInitializer, which using
{{hadoop.http.authentication.kerberos.\***}} ** ** instead of
{{dfs.web.authentication.kerberos}}{{.\}}* to initialize kerberos
# {{hadoop.http.authentication.kerberos.principal}} has a default value, so
even we don't configure this key, the cluster will still start normally
h3. TestRouterHttpDelegationToken
# HDFS-14434 ignores user.name query parameter in secure WebHDFS, and the
initial version of this test leveraged this parameter to bypass the kerberos
authentication, so after HDFS-14434, it's not work. I added a set of methods to
send request by http connection instead of {{WebHdfsFileSystem}} to make it
continue working.
# HADOOP-16314 changed configuration-key of the authentication filter from
{{dfs.web.authentication.filter}} to {{hadoop.http.filter.initializers}}, so I
added an {{NoAuthFilterInitializer}} to initialize {{NoAuthFilter}}
# For case {{testGetDelegationToken()}}, the server address is set by
WebHdfsFileSystem after it get the response, the original address is the
address of RouterRpcServer. Since we now send request by http connection
directly, it's unnecessary to reset the address, so I removed this assert
# For the case {{testCancelDelegationToken()}}, the {{InvalidToken}} exception
is also generated by WebHdfsFileSystem and the logic is very complex, I think
it's also unnecessary to keep this assert, so I using the 403 detection instead.
In the trunk code, the config {{dfs.web.authentication.filter}} is not used
anywhere, I propose to deprecate this config, I'll track this in another Jira.
> RBF: Security should use common AuthenticationFilter
>
>
> Key: HDFS-14609
> URL: https://issues.apache.org/jira/browse/HDFS-14609
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: CR Hota
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14609.001.patch
>
>
> We worked on router based federation security as part of HDFS-13532. We kept
> it compatible with the way namenode works. However with HADOOP-16314 and
> HDFS-16354 in trunk, auth filters seems to have been changed causing tests to
> fail.
> Changes are needed appropriately in RBF, mainly fixing broken tests.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[
https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907486#comment-16907486
]
Chen Zhang edited comment on HDFS-14609 at 8/14/19 5:46 PM:
Thanks [~tasanuma] for providing the old revision of HDFS-13891, it's very
helpful.
I've fixed these 2 tests, here is some detail;
h3. TestRouterWithSecureStartup#testStartupWithoutSpnegoPrincipal
HADOOP-16314 and HADOOP-16354 made some changes which breaks the test:
# Added an AuthFilterInitializer, which using
{{hadoop.http.authentication.kerberos.\***}} ** ** instead of
{{dfs.web.authentication.kerberos}}{{.\}}* to initialize kerberos
# {{hadoop.http.authentication.kerberos.principal}} has a default value, so
even we don't configure this key, the cluster will still start normally
h3. TestRouterHttpDelegationToken
# HDFS-14434 ignores user.name query parameter in secure WebHDFS, and the
initial version of this test leveraged this parameter to bypass the kerberos
authentication, so after HDFS-14434, it's not work. I added a set of methods to
send request by http connection instead of {{WebHdfsFileSystem}} to make it
continue working.
# HADOOP-16314 changed configuration-key of the authentication filter from
{{dfs.web.authentication.filter}} to {{hadoop.http.filter.initializers}}, so I
added an {{NoAuthFilterInitializer}} to initialize {{NoAuthFilter}}
# For case {{testGetDelegationToken()}}, the server address is set by
WebHdfsFileSystem after it get the response, the original address is the
address of RouterRpcServer. Since we now send request by http connection
directly, it's unnecessary to reset the address, so I removed this assert
# For the case {{testCancelDelegationToken()}}, the {{InvalidToken}} exception
is also generated by WebHdfsFileSystem and the logic is very complex, I think
it's also unnecessary to keep this assert, so I using the 403 detection instead.
In the trunk code, the config {{dfs.web.authentication.filter}} is not used
anywhere, I propose to deprecate this config, I'll track this in another Jira.
was (Author: zhangchen):
Thanks [~tasanuma] for providing the old revision of HDFS-13891, it's very
helpful.
I've fixed these 2 tests, here is some detail;
h3. TestRouterWithSecureStartup#testStartupWithoutSpnegoPrincipal
HADOOP-16314 and HADOOP-16354 made some changes which breaks the test:
# Added an AuthFilterInitializer, which using
{{hadoop.http.authentication.kerberos.**}} ** instead of
{{dfs.web.authentication.kerberos}}*{{*.*}}* to initialize kerberos
# {{hadoop.http.authentication.kerberos.principal}} has a default value, so
even we don't configure this key, the cluster will still start normally
h3. TestRouterHttpDelegationToken
# HDFS-14434 ignores user.name query parameter in secure WebHDFS, and the
initial version of this test leveraged this parameter to bypass the kerberos
authentication, so after HDFS-14434, it's not work. I added a set of methods to
send request by http connection instead of {{WebHdfsFileSystem}} to make it
continue working.
# HADOOP-16314 changed configuration-key of the authentication filter from
{{dfs.web.authentication.filter}} to {{hadoop.http.filter.initializers}}, so I
added an {{NoAuthFilterInitializer}} to initialize {{NoAuthFilter}}
# For case {{testGetDelegationToken()}}, the server address is set by
WebHdfsFileSystem after it get the response, the original address is the
address of RouterRpcServer. Since we now send request by http connection
directly, it's unnecessary to reset the address, so I removed this assert
# For the case {{testCancelDelegationToken()}}, the {{InvalidToken}} exception
is also generated by WebHdfsFileSystem and the logic is very complex, I think
it's also unnecessary to keep this assert, so I using the 403 detection instead.
In the trunk code, the config {{dfs.web.authentication.filter}} is not used
anywhere, I propose to deprecate this config, I'll track this in another Jira.
> RBF: Security should use common AuthenticationFilter
>
>
> Key: HDFS-14609
> URL: https://issues.apache.org/jira/browse/HDFS-14609
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: CR Hota
>Assignee: Chen Zhang
>Priority: Major
>
> We worked on router based federation security as part of HDFS-13532. We kept
> it compatible with the way namenode works. However with HADOOP-16314 and
> HDFS-16354 in trunk, auth filters seems to have been changed causing tests to
> fail.
> Changes are needed appropriately in RBF, mainly fixing broken tests.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Comment Edited] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[
https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907486#comment-16907486
]
Chen Zhang edited comment on HDFS-14609 at 8/14/19 5:45 PM:
Thanks [~tasanuma] for providing the old revision of HDFS-13891, it's very
helpful.
I've fixed these 2 tests, here is some detail;
h3. TestRouterWithSecureStartup#testStartupWithoutSpnegoPrincipal
HADOOP-16314 and HADOOP-16354 made some changes which breaks the test:
# Added an AuthFilterInitializer, which using
{{hadoop.http.authentication.kerberos.**}} ** instead of
{{dfs.web.authentication.kerberos}}*{{*.*}}* to initialize kerberos
# {{hadoop.http.authentication.kerberos.principal}} has a default value, so
even we don't configure this key, the cluster will still start normally
h3. TestRouterHttpDelegationToken
# HDFS-14434 ignores user.name query parameter in secure WebHDFS, and the
initial version of this test leveraged this parameter to bypass the kerberos
authentication, so after HDFS-14434, it's not work. I added a set of methods to
send request by http connection instead of {{WebHdfsFileSystem}} to make it
continue working.
# HADOOP-16314 changed configuration-key of the authentication filter from
{{dfs.web.authentication.filter}} to {{hadoop.http.filter.initializers}}, so I
added an {{NoAuthFilterInitializer}} to initialize {{NoAuthFilter}}
# For case {{testGetDelegationToken()}}, the server address is set by
WebHdfsFileSystem after it get the response, the original address is the
address of RouterRpcServer. Since we now send request by http connection
directly, it's unnecessary to reset the address, so I removed this assert
# For the case {{testCancelDelegationToken()}}, the {{InvalidToken}} exception
is also generated by WebHdfsFileSystem and the logic is very complex, I think
it's also unnecessary to keep this assert, so I using the 403 detection instead.
In the trunk code, the config {{dfs.web.authentication.filter}} is not used
anywhere, I propose to deprecate this config, I'll track this in another Jira.
was (Author: zhangchen):
Thanks [~tasanuma] for providing the old revision of HDFS-13891, it's very
helpful.
I've fixed these 2 tests, here is some detail;
h3. TestRouterWithSecureStartup#testStartupWithoutSpnegoPrincipal
HADOOP-16314 and HADOOP-16354 made some changes which breaks the test:
# Added an AuthFilterInitializer, which using
{{hadoop.http.authentication.kerberos.*}} instead of
{{dfs.web.authentication.kerberos.*}} to initialize kerberos
# {{hadoop.http.authentication.kerberos.principal}} has a default value, so
even we don't configure this key, the cluster will still start normally
h3. TestRouterHttpDelegationToken
# HDFS-14434 ignores user.name query parameter in secure WebHDFS, and the
initial version of this test leveraged this parameter to bypass the kerberos
authentication, so after HDFS-14434, it's not work. I added a set of methods to
send request by http connection instead of {{WebHdfsFileSystem}} to make it
continue working.
# HADOOP-16314 changed configuration-key of the authentication filter from
{{dfs.web.authentication.filter}} to {{hadoop.http.filter.initializers}}, so I
added an {{NoAuthFilterInitializer}} to initialize {{NoAuthFilter}}
# For case {{testGetDelegationToken()}}, the server address is set by
WebHdfsFileSystem after it get the response, the original address is the
address of RouterRpcServer. Since we now send request by http connection
directly, it's unnecessary to reset the address, so I removed this assert
# For the case {{testCancelDelegationToken()}}, the {{InvalidToken}} exception
is also generated by WebHdfsFileSystem and the logic is very complex, I think
it's also unnecessary to keep this assert, so I using the 403 detection instead.
In the trunk code, the config {{dfs.web.authentication.filter}} is not used
anywhere, I propose to deprecate this config, I'll track this in another Jira.
> RBF: Security should use common AuthenticationFilter
>
>
> Key: HDFS-14609
> URL: https://issues.apache.org/jira/browse/HDFS-14609
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: CR Hota
>Assignee: Chen Zhang
>Priority: Major
>
> We worked on router based federation security as part of HDFS-13532. We kept
> it compatible with the way namenode works. However with HADOOP-16314 and
> HDFS-16354 in trunk, auth filters seems to have been changed causing tests to
> fail.
> Changes are needed appropriately in RBF, mainly fixing broken tests.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Comment Edited] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[ https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905228#comment-16905228 ] Chen Zhang edited comment on HDFS-14609 at 8/12/19 2:01 PM: Hi [~crh] [~tasanuma], I'm working on this Jira. I found that the branch HDFS-13891 was rebased after HDFS-14074 on trunk, which is committed after HADOOP-16314, so the UT still fail even when we run it on the branch HDFS-13891. I've tried to rebase the HDFS-13891 to some older commit(at 12 Oct 2018), but there are too many conflicts. was (Author: zhangchen): Hi [~crh] [~tasanuma], I'm working on this Jira. I found that the branch HDFS-13891 was rebased after HDFS-14074 on trunk, which is committed after HADOOP-16314, so the UT still fail even when we run it on the branch HDFS-13891. I've tried to rebase the HDFS-13891 to some older commit(at 12 Oct 2018), but there is too many conflicts. > RBF: Security should use common AuthenticationFilter > > > Key: HDFS-14609 > URL: https://issues.apache.org/jira/browse/HDFS-14609 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: CR Hota >Assignee: Chen Zhang >Priority: Major > > We worked on router based federation security as part of HDFS-13532. We kept > it compatible with the way namenode works. However with HADOOP-16314 and > HDFS-16354 in trunk, auth filters seems to have been changed causing tests to > fail. > Changes are needed appropriately in RBF, mainly fixing broken tests. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[
https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873708#comment-16873708
]
Eric Yang edited comment on HDFS-14609 at 6/26/19 11:40 PM:
[~crh]
h3. Answer for TestRouterWithSecureStartup#testStartupWithoutSpnegoPrincipal
issue:
In my local test, the test case continues to fail even when HADOOP-16314 and
HADOOP-16354 are reverted. A few interesting discovery in
[AbstractService.java|https://github.com/apache/hadoop/blame/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/AbstractService.java#L170],
would catch IOException. Hence, the test case expecting IOException, would not
happen. A second issue with the test, it is making assumption that
dfs.web.authentication.kerberos.principal is the principal to run hdfs SPNEGO
principal. However, Hadoop
[KerberosAuthenticationHandler.java|https://github.com/apache/hadoop/blob/d43af8b3db4743b4b240751b6f29de6c20cfd6e5/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java#L133]
uses hadoop.http.authentication.kerberos.principal configuration to obtain
SPNEGO principal. Therefore, no bad configuration was actually detected. This
is a case where too many configurations represent the same thing, and code may
not always work as expected. I am in favor of using
hadoop.http.authentication.kerberos.principal to reduce redundant configuration
keys as the code has written currently.
Here is the unit test log file indicating the HTTP principal is used:
{code:java}
2019-06-26 19:25:25,933 [main] INFO hdfs.DFSUtil
(DFSUtil.java:httpServerTemplateForNNAndJN(1620)) - Starting web server as:
HTTP/localh...@example.com
2019-06-26 19:25:26,215 [main] INFO server.KerberosAuthenticationHandler
(KerberosAuthenticationHandler.java:init(164)) - Using keytab
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/target/test/data/SecurityConfUtil/test.keytab,
for principal HTTP/localh...@example.com
{code}
h3. Answer for TestRouterHttpDelegationToken issue:
The test case NoAuthFilter extends on AuthenticationFilter, which does not
issue delegation token. DelegationTokenAuthenticationFilter is the filter that
issues delegation token. Hence, the test case failed to find delegation token.
I do not know the reason to write test case like this. It seems
counterintuitive to test security feature on a non-secured server.
It looks like RouterHttpServer is extending on top of HttpServer2. Hence, all
the core initialization of security configuration is automatically inherited
from Hadoop. You may need to pay attention to filter initializer to ensure the
security filter is initialized with the one that is expected (configured), and
not hard coded into RouterHttpServer. RouterHttpServer and the two test cases
needs to be refined to make sense.
was (Author: eyang):
[~crh]
h3. Answer for TestRouterWithSecureStartup#testStartupWithoutSpnegoPrincipal
issue:
In my local test, the test case continues to fail even when HADOOP-16314 and
HADOOP-16354 are reverted. A few interesting discovery in
[AbstractService.java|https://github.com/apache/hadoop/blame/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/AbstractService.java#L170],
would catch IOException. Hence, the test case expecting IOException, would not
happen. A second issue with the test, it is making assumption that
dfs.web.authentication.kerberos.principal is the principal to run hdfs SPNEGO
principal. However, Hadoop
[KerberosAuthenticationHandler.java|https://github.com/apache/hadoop/blob/d43af8b3db4743b4b240751b6f29de6c20cfd6e5/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/KerberosAuthenticationHandler.java#L133]
uses hadoop.http.authentication.kerberos.principal configuration to obtain
SPNEGO principal. Therefore, no bad configuration was actually detected. This
is a case where too many configurations represent the same thing, and code may
not always work as expected. I am in favor of using
hadoop.http.authentication.kerberos.principal to reduce redundant configuration
keys as the code has written currently.
Here is the unit test log file indicating the HTTP principal is used:
{code:java}
2019-06-26 19:25:25,933 [main] INFO hdfs.DFSUtil
(DFSUtil.java:httpServerTemplateForNNAndJN(1620)) - Starting web server as:
HTTP/localh...@example.com
2019-06-26 19:25:26,215 [main] INFO server.KerberosAuthenticationHandler
(KerberosAuthenticationHandler.java:init(164)) - Using keytab
/home/eyang/test/hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/target/test/data/SecurityConfUtil/test.keytab,
for principal HTTP/localh...@example.com
{code}
h3. Answer for TestRouterHttpDelegationToken issue:
The test case NoAuthFilter extends on AuthenticationFilter, which does not
