On Tue, Dec 8, 2015 at 4:59 PM, Darpan Patel <[email protected]> wrote:
> Thanks for the merged template. I made modifications to it and > > I am not sure what value should I fill for main.ldapRealm.contextFactory ? > We are running on windows 2008/2012 Active directory. > > <param> > <name>main.ldapRealm.contextFactory</name> > <value>$ldapContextFactory</value> > </param> > > I think that you leave it exactly like that. It is some sort of shiro injection thing - it references the value defined above it that way. > I removed this parameter and I see the in the logs: > > 2015-12-08 21:56:51,806 ERROR hadoop.gateway > (KnoxLdapRealm.java:getUserDn(574)) - Failed to get system ldap connection: > javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: > LdapErr: DSID-0C0903A9, comment: AcceptSecurityContext error, data 52e, > v1db1] > > > ( I am happy to see new error after 3 days phew!!!) > > Glad that you are happy but let's getting working and see how you feel. :) We'll also roll it into some better documentation for the AD specific usecase. > Regards, > DP > > On 8 December 2015 at 14:52, Darpan Patel <[email protected]> wrote: > >> Thanks Larray. >> I will check this and update you. >> >> Regards, >> DP >> >> On 8 December 2015 at 12:18, larry mccay <[email protected]> wrote: >> >>> Hi Darpan - >>> >>> The following topology is probably a better starting point for you AD >>> configuration - I've tried to merge yours with it as best I can: >>> >>> <gateway> >>> <provider> >>> <role>authentication</role> >>> <name>ShiroProvider</name> >>> <enabled>true</enabled> >>> <param> >>> <name>sessionTimeout</name> >>> <value>30</value> >>> </param> >>> <param> >>> <name>main.ldapRealm</name> >>> >>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> >>> </param> >>> >>> <param> >>> <name>main.ldapContextFactory</name> >>> >>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> >>> </param> >>> >>> <param> >>> <name>main.ldapRealm.contextFactory</name> >>> <value>$ldapContextFactory</value> >>> </param> >>> <param> >>> <name>main.ldapRealm.contextFactory.url</name> >>> <!-- ADJUST host, port for your AD setup--> >>> <value>ldap://IP_OF_WINDOWS_AD:389</value> >>> </param> >>> <!-- ignored due to use of >>> main.ldapRealm.userSearchAttributeName --> >>> <param> >>> <name>main.ldapRealm.userDnTemplate</name> >>> <value>cn={0},CN=users,DC=test,DC=com</value> >>> <!-- also tried following values --> >>> <value>uid={0},CN=users,DC=test,DC=com</value> >>> <value>cn={0},DC=test,DC=com</value> >>> </param> >>> >>> <!-- Param above is ignored sAMAccount is usually used for AD --> >>> >>> <param> >>> <name>main.ldapRealm.userSearchAttributeName</name> >>> <value>sAMAccountName</value> >>> </param> >>> >>> <!-- adjust as appropriate --> >>> <param> >>> <name>main.ldapRealm.userObjectClass</name> >>> <value>person</value> >>> </param> >>> >>> <!-- adjust the dn below to match your environment --> >>> <param> >>> <name>main.ldapRealm.contextFactory.systemUsername</name> >>> <value>cn={systemuser},ou=process,ou=accounts,dc=test,dc=com</value> >>> </param> >>> >>> <!-- should be moved to the credential store for the gateway to be more >>> secure --> >>> <param> >>> <name>main.ldapRealm.contextFactory.systemPassword</name> >>> <value>{systemuser_password}/value> >>> </param> >>> >>> <!-- let's disable for now since you have no authorization >>> policies defined anyway --> >>> <param> >>> <name>main.ldapRealm.authorizationEnabled</name> >>> <value>false</value> >>> </param> >>> >>> <param> >>> <name>main.ldapRealm.searchBase</name> >>> <value>cn=users,dc=test,dc=com</value> >>> </param> >>> >>> <param> >>> >>> <param> >>> <name>main.ldapRealm.memberAttributeValueTemplate</name> >>> <value>cn={0},cn=users,dc=test,dc=com</value> >>> <!-- also tried uid={0} --> >>> </param> >>> >>> <param> >>> >>> <name>main.ldapRealm.contextFactory.authenticationMechanism</name> >>> <value>simple</value> >>> </param> >>> >>> <param> >>> <name>urls./**</name> >>> <value>authcBasic</value> >>> </param> >>> </provider> >>> >>> <!-- the group principal mapping below is not likely what you >>> want >>> note that mapping of the hdfs group to admin. Also, we have >>> disabled authorization above so there is no need for >>> groups --> >>> <provider> >>> <role>identity-assertion</role> >>> <name>Default</name> >>> <enabled>true</enabled> >>> <!--param> >>> <name>group.principal.mapping</name> >>> <value>*=users;hdfs=admin</value> >>> </param--> >>> </provider> >>> >>> <provider> >>> <role>authorization</role> >>> <name>AclsAuthz</name> >>> <enabled>true</enabled> >>> </provider> >>> >>> </gateway> >>> >>> We need to better document the difference between LDAP and AD for such >>> deployments. >>> >>> I've also tried to document some of the changes that I made. >>> Note that you don't have any authorization ACLs defined in the AclsAuthz >>> provider so I disabled group lookup. >>> That will only add complexity to your config - we can re-enable once >>> authentication is working. >>> >>> Please go through this config and ensure that DNs, host and ports and >>> system usernames match your environment. >>> >>> Hope this helps. >>> >>> --larry >>> >>> On Tue, Dec 8, 2015 at 5:16 AM, Darpan Patel <[email protected]> wrote: >>> >>>> Hi All, >>>> >>>> For this blocker issue let more information if it can help fixing the >>>> authorization problem. >>>> Please let me know if more details required. >>>> (+ dev list) >>>> >>>> */etc/krb5.conf* >>>> >>>> [libdefaults] >>>> renew_lifetime = 7d >>>> forwardable = true >>>> default_realm = HORTONWORKS.COM >>>> ticket_lifetime = 24h >>>> dns_lookup_realm = false >>>> dns_lookup_kdc = false >>>> #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 >>>> #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 >>>> >>>> [domain_realm] >>>> .hortonworks.com = HORTONWORKS.COM >>>> HORTONWORKS.COm = HORTONWORKS.COM >>>> >>>> [logging] >>>> default = FILE:/var/log/krb5kdc.log >>>> admin_server = FILE:/var/log/kadmind.log >>>> kdc = FILE:/var/log/krb5kdc.log >>>> >>>> [realms] >>>> HORTONWORKS.COM = { >>>> admin_server = KDC_SERVER_HOST >>>> kdc = KDC_SERVER_HOST >>>> } >>>> *TEST.COM <http://TEST.COM>* = { >>>> admin_server = WINDOWS_12_SERVER_AD_HOST >>>> kdc = WINDOWS_12_SERVER_AD_HOST >>>> } >>>> >>>> >>>> */usr/hdp/current/knox-server/conf/gateway-site.xml* >>>> >>>> <configuration> >>>> <property> >>>> <name>*gateway.gateway.conf.dir*</name> >>>> <value>deployments</value> >>>> </property> >>>> <property> >>>> <name>*gateway.hadoop.kerberos.secured*</name> >>>> <value>true</value> >>>> </property> >>>> <property> >>>> <name>*gateway.path*</name> >>>> <value>gateway</value> >>>> </property> >>>> <property> >>>> <name>*gateway.port*</name> >>>> <value>8443</value> >>>> </property> >>>> <property> >>>> <name>*java.security.auth.login.config*</name> >>>> <value>/*etc/knox/conf/krb5JAASLogin.conf*</value> >>>> </property> >>>> <property> >>>> <name>*java.security.krb5.conf*</name> >>>> <value>*/etc/krb5.conf*</value> >>>> </property> >>>> <property> >>>> <name>sun.security.krb5.debug</name> >>>> <value>true</value> >>>> </property> >>>> </configuration> >>>> >>>> >>>> */etc/knox/conf/krb5JAASLogin.conf* >>>> >>>> com.sun.security.jgss.initiate { >>>> com.sun.security.auth.module.Krb5LoginModule required >>>> renewTGT=true >>>> doNotPrompt=true >>>> useKeyTab=true >>>> keyTab="/etc/security/keytabs/knox.service.keytab" >>>> principal="knox/[email protected]" >>>> isInitiator=true >>>> storeKey=true >>>> useTicketCache=true >>>> client=true; >>>> }; >>>> >>>> Regards, >>>> DP >>>> >>>> ---------- Forwarded message ---------- >>>> From: Darpan Patel <[email protected]> >>>> Date: 7 December 2015 at 17:59 >>>> Subject: Need help setting up Knox for A/D integrated Kerberized Cluster >>>> To: [email protected] >>>> >>>> >>>> Hi All, >>>> >>>> I am stuck on an issue from last two days. I would be really grateful >>>> if someone can help on this. >>>> >>>> We have HDP 2.3 implemented over 8 node cluster and the same cluster >>>> has been Kerberized and later on we have integrated it with Active >>>> Directory (Which runs in the same VPN). We also verified that Windows 2012 >>>> A/D integration with Ranger works fine for defining policies and audit log. >>>> But I am stuck at Knox bit. I am trying to replicate the same configuration >>>> properties which I have set for Ranger LDAP-AD Integration. >>>> >>>> I am taking reference of the Hortonworks documentation and also Apache >>>> Knox documentation. >>>> >>>> The A/D domain name is TEST.COM and all the users are under Users >>>> >>>> [image: Inline images 1] >>>> >>>> >>>> Under the Users we have few users one of the them is knox, darpan, >>>> test,etc. >>>> >>>> When we issue following command on the node on which Knox Server is >>>> running (topology name is default) >>>> >>>> >>>> >>>> *curl -iv -k -u [email protected]:#123Password -X GET >>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS >>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>" OR* >>>> >>>> *curl -iv -k -u knox:#123Password -X GET >>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS >>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>"* >>>> >>>> >>>> Every time I see < HTTP/1.1 401 Unauthorized HTTP/1.1 401 Unauthorized >>>> on the console. >>>> >>>> >>>> Entries in the *gateway-audit.log *are like this : >>>> >>>> gateway-audit.log >>>> ================== >>>> 15/12/07 17:11:08 >>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable| >>>> 15/12/07 17:11:09 >>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||authentication| >>>> *principal*|*[email protected] <[email protected]>*|failure|*LDAP >>>> authentication failed.* >>>> 15/12/07 17:11:09 >>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response >>>> status: 401 >>>> >>>> >>>> 15/12/07 17:05:28 >>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable| >>>> 15/12/07 17:05:29 >>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||authentication| >>>> *principal*|knox|failure|*LDAP authentication failed.* >>>> 15/12/07 17:05:29 >>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response >>>> status: 401 >>>> >>>> >>>> *Gateway.log* >>>> *===========* >>>> 2015-12-07 17:05:28,620 INFO hadoop.gateway >>>> (KnoxLdapRealm.java:getUserDn(550)) - Computed userDn: >>>> cn=knox,CN=users,DC=test,DC=com using dnTemplate for principal: knox >>>> >>>> >>>> Following is the part of our *default.xml *topology: >>>> >>>> >>>> <gateway> >>>> <provider> >>>> <role>authentication</role> >>>> <name>ShiroProvider</name> >>>> <enabled>true</enabled> >>>> <param> >>>> <name>sessionTimeout</name> >>>> <value>30</value> >>>> </param> >>>> <param> >>>> <name>*main.ldapRealm*</name> >>>> >>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> >>>> </param> >>>> >>>> <param> >>>> <name>*main.ldapContextFactory*</name> >>>> >>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value> >>>> </param> >>>> >>>> <param> >>>> <name>*main.ldapRealm.userDnTemplate*</name> >>>> <value>cn={0},CN=users,DC=test,DC=com</value> >>>> <!-- also tried following values --> >>>> <value>uid={0},CN=users,DC=test,DC=com</value> >>>> <value>cn={0},DC=test,DC=com</value> >>>> </param> >>>> <param> >>>> <name>*main.ldapRealm.contextFactory.url* >>>> </name> >>>> <!-- IP Address of the WINDOSWS 2012 Acive >>>> Directory Server which works for Ranger --> >>>> <value>*ldap://IP_OF_WINDOWS_AD:389*</value> >>>> </param> >>>> <param> >>>> <name>*main.ldapRealm.authorizationEnabled* >>>> </name> >>>> <value>true</value> >>>> </param> >>>> <param> >>>> <name>*main.ldapRealm.searchBase*</name> >>>> <value>cn=users,dc=test,dc=com</value> >>>> </param> >>>> <param> >>>> <param> >>>> <name> >>>> *main.ldapRealm.memberAttributeValueTemplate*</name> >>>> <value>cn={0},cn=users,dc=test,dc=com</value> >>>> <!-- also tried uid={0} --> >>>> </param> >>>> <param> >>>> <name> >>>> *main.ldapRealm.contextFactory.authenticationMechanism<*/name> >>>> <value>simple</value> >>>> </param> >>>> <param> >>>> <name>urls./**</name> >>>> <value>authcBasic</value> >>>> </param> >>>> </provider> >>>> >>>> <provider> >>>> <role>*identity-assertion*</role> >>>> <name>Default</name> >>>> <enabled>true</enabled> >>>> <param> >>>> <name>*group.principal.mapping*</name> >>>> <value>*=users;hdfs=admin</value> >>>> </param> >>>> </provider> >>>> >>>> <provider> >>>> <role>*authorization*</role> >>>> <name>AclsAuthz</name> >>>> <enabled>true</enabled> >>>> </provider> >>>> >>>> </gateway> >>>> >>>> >>>> And following is the console output while trying to access webhdfs >>>> using curl >>>> >>>> curl -iv -k -u knox:#123Password -X GET " >>>> https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS"; >>>> >>>> >>>> *Console Output:* >>>> ---------------- >>>> >>>> * About to connect() to localhost port 8443 (#0) >>>> * Trying ::1... >>>> * Connected to localhost (::1) port 8443 (#0) >>>> * Initializing NSS with certpath: sql:/etc/pki/nssdb >>>> * skipping SSL peer certificate verification >>>> * SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA >>>> * Server certificate: >>>> * subject: >>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US >>>> * start date: Nov 27 20:36:22 2015 GMT >>>> * expire date: Nov 26 20:36:22 2016 GMT >>>> * common name: FQDN_OF_My_gateway_HOST >>>> * issuer: >>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US >>>> * Server auth using Basic with user 'knox' >>>> > GET /gateway/default/webhdfs/v1/?op=LISTSTATUS HTTP/1.1 >>>> > Authorization: Basic a25veDojMTIzUGFzc3dvcmQ= >>>> > User-Agent: curl/7.29.0 >>>> > Host: localhost:8443 >>>> > Accept: */* >>>> > >>>> < HTTP/1.1 401 Unauthorized >>>> HTTP/1.1 401 Unauthorized >>>> * Authentication problem. Ignoring this. >>>> < WWW-Authenticate: BASIC realm="application" >>>> WWW-Authenticate: BASIC realm="application" >>>> < Content-Length: 0 >>>> Content-Length: 0 >>>> < Server: Jetty(8.1.14.v20131031) >>>> Server: Jetty(8.1.14.v20131031) >>>> >>>> >>>> Please let me know if any additional information is required. >>>> >>>> Thanks, >>>> DP >>>> >>>> >>> >> >
