[jira] [Commented] (HDFS-14525) JspHelper ignores hadoop.http.authentication.type
[ https://issues.apache.org/jira/browse/HDFS-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327104#comment-17327104 ] Qi Zhu commented on HDFS-14525: --- [~prabhujoseph] I also think this is needed, if we can add an option to support: We can add an option to allow this two independent, hadoop.security.authentication is specific to RPC Authentication whereas hadoop.http.authentication.type is specific to HTTP Authentication. We want to make HTTP not authentication, but RPC Authentication. > JspHelper ignores hadoop.http.authentication.type > - > > Key: HDFS-14525 > URL: https://issues.apache.org/jira/browse/HDFS-14525 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Priority: Major > > On Secure Cluster With hadoop.http.authentication.type simple and > hadoop.http.authentication.anonymous.allowed is true, WebHdfs Rest Api fails > when user.name is not set. It runs fine if user.name=ambari-qa is set.. > {code} > [knox@pjosephdocker-1 ~]$ curl -sS -L -w '%{http_code}' -X GET -d '' -H > 'Content-Length: 0' --negotiate -u : > 'http://pjosephdocker-1.openstacklocal:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS' > {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed > to obtain user group information: java.io.IOException: Security enabled but > user not authenticated by filter"}}403[knox@pjosephdocker-1 ~]$ > {code} > JspHelper#getUGI checks UserGroupInformation.isSecurityEnabled() instead of > conf.get(hadoop.http.authentication.type).equals("kerberos") to check if Http > is Secure causing the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14525) JspHelper ignores hadoop.http.authentication.type
[ https://issues.apache.org/jira/browse/HDFS-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853253#comment-16853253 ] Prabhu Joseph commented on HDFS-14525: -- Thanks [~eyang] for the clarifications. Will close this Jira as Working as designed. > JspHelper ignores hadoop.http.authentication.type > - > > Key: HDFS-14525 > URL: https://issues.apache.org/jira/browse/HDFS-14525 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Priority: Major > > On Secure Cluster With hadoop.http.authentication.type simple and > hadoop.http.authentication.anonymous.allowed is true, WebHdfs Rest Api fails > when user.name is not set. It runs fine if user.name=ambari-qa is set.. > {code} > [knox@pjosephdocker-1 ~]$ curl -sS -L -w '%{http_code}' -X GET -d '' -H > 'Content-Length: 0' --negotiate -u : > 'http://pjosephdocker-1.openstacklocal:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS' > {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed > to obtain user group information: java.io.IOException: Security enabled but > user not authenticated by filter"}}403[knox@pjosephdocker-1 ~]$ > {code} > JspHelper#getUGI checks UserGroupInformation.isSecurityEnabled() instead of > conf.get(hadoop.http.authentication.type).equals("kerberos") to check if Http > is Secure causing the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14525) JspHelper ignores hadoop.http.authentication.type
[ https://issues.apache.org/jira/browse/HDFS-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853220#comment-16853220 ] Eric Yang commented on HDFS-14525: -- [~Prabhu Joseph] {quote}JspHelper fails Anonymous user requests even though the http request is successfully authenticated by PseudoAuthenticationHandler.{quote} Other type of authentication handler is a valid point. I do not know if any Hadoop down stream client that actually implements Kerberos RPC + LDAP HTTP had any success, but the possibility of trying is real. It would be tough to ask them to simplify the security model, if they already trying this combo. However, given how UGI works, the Login user must have a kerberos ticket associated with the user in Kerberos environment. PseudoAuthenticationHandler will generate a anonymous token without kerberos ticket, and the token can not be verified with KDC. By allowing PseudoAuthenticationHandler anonymous token to pass through, can create a loophole in Hadoop security. This is not allowed to prevent anonymous token to roam freely in Hadoop cluster. This is the reason that it looks correct to change code to use conf.get(hadoop.http.authentication.type)!=SIMPLE for web protocol, but it is not exactly safe, given how UGI works. > JspHelper ignores hadoop.http.authentication.type > - > > Key: HDFS-14525 > URL: https://issues.apache.org/jira/browse/HDFS-14525 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Priority: Major > > On Secure Cluster With hadoop.http.authentication.type simple and > hadoop.http.authentication.anonymous.allowed is true, WebHdfs Rest Api fails > when user.name is not set. It runs fine if user.name=ambari-qa is set.. > {code} > [knox@pjosephdocker-1 ~]$ curl -sS -L -w '%{http_code}' -X GET -d '' -H > 'Content-Length: 0' --negotiate -u : > 'http://pjosephdocker-1.openstacklocal:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS' > {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed > to obtain user group information: java.io.IOException: Security enabled but > user not authenticated by filter"}}403[knox@pjosephdocker-1 ~]$ > {code} > JspHelper#getUGI checks UserGroupInformation.isSecurityEnabled() instead of > conf.get(hadoop.http.authentication.type).equals("kerberos") to check if Http > is Secure causing the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14525) JspHelper ignores hadoop.http.authentication.type
[ https://issues.apache.org/jira/browse/HDFS-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852854#comment-16852854 ] Prabhu Joseph commented on HDFS-14525: -- [~eyang] Thanks for the inputs. 1. As per my understanding, hadoop.security.authentication is specific to RPC Authentication whereas hadoop.http.authentication.type is specific to HTTP Authentication. We have simple, kerberos authentication for RPC whereas HTTP Authentication can be simple, kerberos, ldap (LdapAuthenticationHandler), WebSSO (JWTRedirectAuthenticationHandler) which is used for Knox and also can be custom. Customers uses ldap or websso for HTTP and kerberos for RPC. I think we need a separate config for HTTP to have different authentication behavior from RPC. And as per the testing, fixing below two places should be fine. 1. HttpServer2 does Kerberos initSpnego when hadoop.security.authentication is kerberos which can cause http requests (Pseudo) failing with "Authentication Required". Will fix this in Hadoop-16314. 2. JspHelper fails Anonymous user requests even though the http request is successfully authenticated by PseudoAuthenticationHandler. Please let me know how to proceed further. Thanks. > JspHelper ignores hadoop.http.authentication.type > - > > Key: HDFS-14525 > URL: https://issues.apache.org/jira/browse/HDFS-14525 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Priority: Major > > On Secure Cluster With hadoop.http.authentication.type simple and > hadoop.http.authentication.anonymous.allowed is true, WebHdfs Rest Api fails > when user.name is not set. It runs fine if user.name=ambari-qa is set.. > {code} > [knox@pjosephdocker-1 ~]$ curl -sS -L -w '%{http_code}' -X GET -d '' -H > 'Content-Length: 0' --negotiate -u : > 'http://pjosephdocker-1.openstacklocal:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS' > {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed > to obtain user group information: java.io.IOException: Security enabled but > user not authenticated by filter"}}403[knox@pjosephdocker-1 ~]$ > {code} > JspHelper#getUGI checks UserGroupInformation.isSecurityEnabled() instead of > conf.get(hadoop.http.authentication.type).equals("kerberos") to check if Http > is Secure causing the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14525) JspHelper ignores hadoop.http.authentication.type
[ https://issues.apache.org/jira/browse/HDFS-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852447#comment-16852447 ] Eric Yang commented on HDFS-14525: -- [~daryn], [~Prabhu Joseph] This boils down to if there is any valid use case to keep hadoop.http.authentication.type independent of hadoop.security.authentication? In various Hadoop code, there are inter-exchange of using UserGroupInformation.isSecurityEnabled() for web protocol, for example in [DFSUtil.java|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java#L1614]. This mistake indicates that developers have the interests in heart to standardize the use of hadoop.security.authentication setting and keep anonymous user out. Maybe it is time to revisit if there is valid use case to set hadoop.http.authentication.type differently from hadoop.security.authentication? If there is no valid use case, then we probably want to deprecate hadoop.http.authentication.type to avoid the circular discussions. All web app can depend on UserGroupInformation.isSecurityEnabled(). Hence, this bug can be invalided. My vote is to deprecate hadoop.http.authentication.type setting to avoid code incorrectness and confusions. > JspHelper ignores hadoop.http.authentication.type > - > > Key: HDFS-14525 > URL: https://issues.apache.org/jira/browse/HDFS-14525 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Priority: Major > > On Secure Cluster With hadoop.http.authentication.type simple and > hadoop.http.authentication.anonymous.allowed is true, WebHdfs Rest Api fails > when user.name is not set. It runs fine if user.name=ambari-qa is set.. > {code} > [knox@pjosephdocker-1 ~]$ curl -sS -L -w '%{http_code}' -X GET -d '' -H > 'Content-Length: 0' --negotiate -u : > 'http://pjosephdocker-1.openstacklocal:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS' > {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed > to obtain user group information: java.io.IOException: Security enabled but > user not authenticated by filter"}}403[knox@pjosephdocker-1 ~]$ > {code} > JspHelper#getUGI checks UserGroupInformation.isSecurityEnabled() instead of > conf.get(hadoop.http.authentication.type).equals("kerberos") to check if Http > is Secure causing the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14525) JspHelper ignores hadoop.http.authentication.type
[ https://issues.apache.org/jira/browse/HDFS-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852267#comment-16852267 ] Prabhu Joseph commented on HDFS-14525: -- bq. You actually want a secure cluster to accept anonymous users? Why do you even have security enabled? Then why we have a separate config hadoop.http.authentication.simple.anonymous.allowed which adds complexity in testing all the scenarios while making new changes. Yes the proposed change is wrong. I think the below will work. {code} UserGroupInformation.isSecurityEnabled() && !conf.get(hadoop.http.authentication.type).equals("simple") {code} > JspHelper ignores hadoop.http.authentication.type > - > > Key: HDFS-14525 > URL: https://issues.apache.org/jira/browse/HDFS-14525 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Priority: Major > > On Secure Cluster With hadoop.http.authentication.type simple and > hadoop.http.authentication.anonymous.allowed is true, WebHdfs Rest Api fails > when user.name is not set. It runs fine if user.name=ambari-qa is set.. > {code} > [knox@pjosephdocker-1 ~]$ curl -sS -L -w '%{http_code}' -X GET -d '' -H > 'Content-Length: 0' --negotiate -u : > 'http://pjosephdocker-1.openstacklocal:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS' > {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed > to obtain user group information: java.io.IOException: Security enabled but > user not authenticated by filter"}}403[knox@pjosephdocker-1 ~]$ > {code} > JspHelper#getUGI checks UserGroupInformation.isSecurityEnabled() instead of > conf.get(hadoop.http.authentication.type).equals("kerberos") to check if Http > is Secure causing the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14525) JspHelper ignores hadoop.http.authentication.type
[ https://issues.apache.org/jira/browse/HDFS-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852255#comment-16852255 ] Daryn Sharp commented on HDFS-14525: You actually want a secure cluster to accept anonymous users? Why do you even have security enabled? I believe there was a debate about anon with security many years ago and the decision was anon is fundamentally incompatible with security. The proposed change is completely wrong. If a custom authentication method is used, you cannot fall through and allow the client specify who they are. I'd say this is working as designed. > JspHelper ignores hadoop.http.authentication.type > - > > Key: HDFS-14525 > URL: https://issues.apache.org/jira/browse/HDFS-14525 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Priority: Major > > On Secure Cluster With hadoop.http.authentication.type simple and > hadoop.http.authentication.anonymous.allowed is true, WebHdfs Rest Api fails > when user.name is not set. It runs fine if user.name=ambari-qa is set.. > {code} > [knox@pjosephdocker-1 ~]$ curl -sS -L -w '%{http_code}' -X GET -d '' -H > 'Content-Length: 0' --negotiate -u : > 'http://pjosephdocker-1.openstacklocal:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS' > {"RemoteException":{"exception":"SecurityException","javaClassName":"java.lang.SecurityException","message":"Failed > to obtain user group information: java.io.IOException: Security enabled but > user not authenticated by filter"}}403[knox@pjosephdocker-1 ~]$ > {code} > JspHelper#getUGI checks UserGroupInformation.isSecurityEnabled() instead of > conf.get(hadoop.http.authentication.type).equals("kerberos") to check if Http > is Secure causing the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org