Thanks for the merged template. I made modifications to it and
I am not sure what value should I fill for main.ldapRealm.contextFactory ?
We are running on windows 2008/2012 Active directory.
<param>
<name>main.ldapRealm.contextFactory</name>
<value>$ldapContextFactory</value>
</param>
I removed this parameter and I see the in the logs:
2015-12-08 21:56:51,806 ERROR hadoop.gateway
(KnoxLdapRealm.java:getUserDn(574)) - Failed to get system ldap connection:
javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308:
LdapErr: DSID-0C0903A9, comment: AcceptSecurityContext error, data 52e,
v1db1]
( I am happy to see new error after 3 days phew!!!)
Regards,
DP
On 8 December 2015 at 14:52, Darpan Patel <[email protected]> wrote:
> Thanks Larray.
> I will check this and update you.
>
> Regards,
> DP
>
> On 8 December 2015 at 12:18, larry mccay <[email protected]> wrote:
>
>> Hi Darpan -
>>
>> The following topology is probably a better starting point for you AD
>> configuration - I've tried to merge yours with it as best I can:
>>
>> <gateway>
>> <provider>
>> <role>authentication</role>
>> <name>ShiroProvider</name>
>> <enabled>true</enabled>
>> <param>
>> <name>sessionTimeout</name>
>> <value>30</value>
>> </param>
>> <param>
>> <name>main.ldapRealm</name>
>>
>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
>> </param>
>>
>> <param>
>> <name>main.ldapContextFactory</name>
>>
>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
>> </param>
>>
>> <param>
>> <name>main.ldapRealm.contextFactory</name>
>> <value>$ldapContextFactory</value>
>> </param>
>> <param>
>> <name>main.ldapRealm.contextFactory.url</name>
>> <!-- ADJUST host, port for your AD setup-->
>> <value>ldap://IP_OF_WINDOWS_AD:389</value>
>> </param>
>> <!-- ignored due to use of
>> main.ldapRealm.userSearchAttributeName -->
>> <param>
>> <name>main.ldapRealm.userDnTemplate</name>
>> <value>cn={0},CN=users,DC=test,DC=com</value>
>> <!-- also tried following values -->
>> <value>uid={0},CN=users,DC=test,DC=com</value>
>> <value>cn={0},DC=test,DC=com</value>
>> </param>
>>
>> <!-- Param above is ignored sAMAccount is usually used for AD -->
>>
>> <param>
>> <name>main.ldapRealm.userSearchAttributeName</name>
>> <value>sAMAccountName</value>
>> </param>
>>
>> <!-- adjust as appropriate -->
>> <param>
>> <name>main.ldapRealm.userObjectClass</name>
>> <value>person</value>
>> </param>
>>
>> <!-- adjust the dn below to match your environment -->
>> <param>
>> <name>main.ldapRealm.contextFactory.systemUsername</name>
>> <value>cn={systemuser},ou=process,ou=accounts,dc=test,dc=com</value>
>> </param>
>>
>> <!-- should be moved to the credential store for the gateway to be more
>> secure -->
>> <param>
>> <name>main.ldapRealm.contextFactory.systemPassword</name>
>> <value>{systemuser_password}/value>
>> </param>
>>
>> <!-- let's disable for now since you have no authorization
>> policies defined anyway -->
>> <param>
>> <name>main.ldapRealm.authorizationEnabled</name>
>> <value>false</value>
>> </param>
>>
>> <param>
>> <name>main.ldapRealm.searchBase</name>
>> <value>cn=users,dc=test,dc=com</value>
>> </param>
>>
>> <param>
>>
>> <param>
>> <name>main.ldapRealm.memberAttributeValueTemplate</name>
>> <value>cn={0},cn=users,dc=test,dc=com</value>
>> <!-- also tried uid={0} -->
>> </param>
>>
>> <param>
>>
>> <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
>> <value>simple</value>
>> </param>
>>
>> <param>
>> <name>urls./**</name>
>> <value>authcBasic</value>
>> </param>
>> </provider>
>>
>> <!-- the group principal mapping below is not likely what you
>> want
>> note that mapping of the hdfs group to admin. Also, we have
>> disabled authorization above so there is no need for groups
>> -->
>> <provider>
>> <role>identity-assertion</role>
>> <name>Default</name>
>> <enabled>true</enabled>
>> <!--param>
>> <name>group.principal.mapping</name>
>> <value>*=users;hdfs=admin</value>
>> </param-->
>> </provider>
>>
>> <provider>
>> <role>authorization</role>
>> <name>AclsAuthz</name>
>> <enabled>true</enabled>
>> </provider>
>>
>> </gateway>
>>
>> We need to better document the difference between LDAP and AD for such
>> deployments.
>>
>> I've also tried to document some of the changes that I made.
>> Note that you don't have any authorization ACLs defined in the AclsAuthz
>> provider so I disabled group lookup.
>> That will only add complexity to your config - we can re-enable once
>> authentication is working.
>>
>> Please go through this config and ensure that DNs, host and ports and
>> system usernames match your environment.
>>
>> Hope this helps.
>>
>> --larry
>>
>> On Tue, Dec 8, 2015 at 5:16 AM, Darpan Patel <[email protected]> wrote:
>>
>>> Hi All,
>>>
>>> For this blocker issue let more information if it can help fixing the
>>> authorization problem.
>>> Please let me know if more details required.
>>> (+ dev list)
>>>
>>> */etc/krb5.conf*
>>>
>>> [libdefaults]
>>> renew_lifetime = 7d
>>> forwardable = true
>>> default_realm = HORTONWORKS.COM
>>> ticket_lifetime = 24h
>>> dns_lookup_realm = false
>>> dns_lookup_kdc = false
>>> #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
>>> #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
>>>
>>> [domain_realm]
>>> .hortonworks.com = HORTONWORKS.COM
>>> HORTONWORKS.COm = HORTONWORKS.COM
>>>
>>> [logging]
>>> default = FILE:/var/log/krb5kdc.log
>>> admin_server = FILE:/var/log/kadmind.log
>>> kdc = FILE:/var/log/krb5kdc.log
>>>
>>> [realms]
>>> HORTONWORKS.COM = {
>>> admin_server = KDC_SERVER_HOST
>>> kdc = KDC_SERVER_HOST
>>> }
>>> *TEST.COM <http://TEST.COM>* = {
>>> admin_server = WINDOWS_12_SERVER_AD_HOST
>>> kdc = WINDOWS_12_SERVER_AD_HOST
>>> }
>>>
>>>
>>> */usr/hdp/current/knox-server/conf/gateway-site.xml*
>>>
>>> <configuration>
>>> <property>
>>> <name>*gateway.gateway.conf.dir*</name>
>>> <value>deployments</value>
>>> </property>
>>> <property>
>>> <name>*gateway.hadoop.kerberos.secured*</name>
>>> <value>true</value>
>>> </property>
>>> <property>
>>> <name>*gateway.path*</name>
>>> <value>gateway</value>
>>> </property>
>>> <property>
>>> <name>*gateway.port*</name>
>>> <value>8443</value>
>>> </property>
>>> <property>
>>> <name>*java.security.auth.login.config*</name>
>>> <value>/*etc/knox/conf/krb5JAASLogin.conf*</value>
>>> </property>
>>> <property>
>>> <name>*java.security.krb5.conf*</name>
>>> <value>*/etc/krb5.conf*</value>
>>> </property>
>>> <property>
>>> <name>sun.security.krb5.debug</name>
>>> <value>true</value>
>>> </property>
>>> </configuration>
>>>
>>>
>>> */etc/knox/conf/krb5JAASLogin.conf*
>>>
>>> com.sun.security.jgss.initiate {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> renewTGT=true
>>> doNotPrompt=true
>>> useKeyTab=true
>>> keyTab="/etc/security/keytabs/knox.service.keytab"
>>> principal="knox/[email protected]"
>>> isInitiator=true
>>> storeKey=true
>>> useTicketCache=true
>>> client=true;
>>> };
>>>
>>> Regards,
>>> DP
>>>
>>> ---------- Forwarded message ----------
>>> From: Darpan Patel <[email protected]>
>>> Date: 7 December 2015 at 17:59
>>> Subject: Need help setting up Knox for A/D integrated Kerberized Cluster
>>> To: [email protected]
>>>
>>>
>>> Hi All,
>>>
>>> I am stuck on an issue from last two days. I would be really grateful if
>>> someone can help on this.
>>>
>>> We have HDP 2.3 implemented over 8 node cluster and the same cluster has
>>> been Kerberized and later on we have integrated it with Active Directory
>>> (Which runs in the same VPN). We also verified that Windows 2012 A/D
>>> integration with Ranger works fine for defining policies and audit log. But
>>> I am stuck at Knox bit. I am trying to replicate the same configuration
>>> properties which I have set for Ranger LDAP-AD Integration.
>>>
>>> I am taking reference of the Hortonworks documentation and also Apache
>>> Knox documentation.
>>>
>>> The A/D domain name is TEST.COM and all the users are under Users
>>>
>>> [image: Inline images 1]
>>>
>>>
>>> Under the Users we have few users one of the them is knox, darpan,
>>> test,etc.
>>>
>>> When we issue following command on the node on which Knox Server is
>>> running (topology name is default)
>>>
>>>
>>>
>>> *curl -iv -k -u [email protected]:#123Password -X GET
>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS
>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>" OR*
>>>
>>> *curl -iv -k -u knox:#123Password -X GET
>>> "https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS
>>> <https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS>"*
>>>
>>>
>>> Every time I see < HTTP/1.1 401 Unauthorized HTTP/1.1 401 Unauthorized
>>> on the console.
>>>
>>>
>>> Entries in the *gateway-audit.log *are like this :
>>>
>>> gateway-audit.log
>>> ==================
>>> 15/12/07 17:11:08
>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable|
>>> 15/12/07 17:11:09
>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||authentication|
>>> *principal*|*[email protected] <[email protected]>*|failure|*LDAP
>>> authentication failed.*
>>> 15/12/07 17:11:09
>>> ||38606993-17e2-4c3e-ad4b-e3faea293aae|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response
>>> status: 401
>>>
>>>
>>> 15/12/07 17:05:28
>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|unavailable|
>>> 15/12/07 17:05:29
>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||authentication|
>>> *principal*|knox|failure|*LDAP authentication failed.*
>>> 15/12/07 17:05:29
>>> ||5b436e43-b874-40f7-b111-7b262fe5125d|audit|WEBHDFS||||access|uri|/gateway/default/webhdfs/v1/?op=LISTSTATUS|success|Response
>>> status: 401
>>>
>>>
>>> *Gateway.log*
>>> *===========*
>>> 2015-12-07 17:05:28,620 INFO hadoop.gateway
>>> (KnoxLdapRealm.java:getUserDn(550)) - Computed userDn:
>>> cn=knox,CN=users,DC=test,DC=com using dnTemplate for principal: knox
>>>
>>>
>>> Following is the part of our *default.xml *topology:
>>>
>>>
>>> <gateway>
>>> <provider>
>>> <role>authentication</role>
>>> <name>ShiroProvider</name>
>>> <enabled>true</enabled>
>>> <param>
>>> <name>sessionTimeout</name>
>>> <value>30</value>
>>> </param>
>>> <param>
>>> <name>*main.ldapRealm*</name>
>>>
>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
>>> </param>
>>>
>>> <param>
>>> <name>*main.ldapContextFactory*</name>
>>>
>>> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
>>> </param>
>>>
>>> <param>
>>> <name>*main.ldapRealm.userDnTemplate*</name>
>>> <value>cn={0},CN=users,DC=test,DC=com</value>
>>> <!-- also tried following values -->
>>> <value>uid={0},CN=users,DC=test,DC=com</value>
>>> <value>cn={0},DC=test,DC=com</value>
>>> </param>
>>> <param>
>>> <name>*main.ldapRealm.contextFactory.url*</name>
>>> <!-- IP Address of the WINDOSWS 2012 Acive
>>> Directory Server which works for Ranger -->
>>> <value>*ldap://IP_OF_WINDOWS_AD:389*</value>
>>> </param>
>>> <param>
>>> <name>*main.ldapRealm.authorizationEnabled*</name>
>>> <value>true</value>
>>> </param>
>>> <param>
>>> <name>*main.ldapRealm.searchBase*</name>
>>> <value>cn=users,dc=test,dc=com</value>
>>> </param>
>>> <param>
>>> <param>
>>> <name>
>>> *main.ldapRealm.memberAttributeValueTemplate*</name>
>>> <value>cn={0},cn=users,dc=test,dc=com</value>
>>> <!-- also tried uid={0} -->
>>> </param>
>>> <param>
>>> <name>
>>> *main.ldapRealm.contextFactory.authenticationMechanism<*/name>
>>> <value>simple</value>
>>> </param>
>>> <param>
>>> <name>urls./**</name>
>>> <value>authcBasic</value>
>>> </param>
>>> </provider>
>>>
>>> <provider>
>>> <role>*identity-assertion*</role>
>>> <name>Default</name>
>>> <enabled>true</enabled>
>>> <param>
>>> <name>*group.principal.mapping*</name>
>>> <value>*=users;hdfs=admin</value>
>>> </param>
>>> </provider>
>>>
>>> <provider>
>>> <role>*authorization*</role>
>>> <name>AclsAuthz</name>
>>> <enabled>true</enabled>
>>> </provider>
>>>
>>> </gateway>
>>>
>>>
>>> And following is the console output while trying to access webhdfs using
>>> curl
>>>
>>> curl -iv -k -u knox:#123Password -X GET "
>>> https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS";
>>>
>>>
>>> *Console Output:*
>>> ----------------
>>>
>>> * About to connect() to localhost port 8443 (#0)
>>> * Trying ::1...
>>> * Connected to localhost (::1) port 8443 (#0)
>>> * Initializing NSS with certpath: sql:/etc/pki/nssdb
>>> * skipping SSL peer certificate verification
>>> * SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA
>>> * Server certificate:
>>> * subject:
>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
>>> * start date: Nov 27 20:36:22 2015 GMT
>>> * expire date: Nov 26 20:36:22 2016 GMT
>>> * common name: FQDN_OF_My_gateway_HOST
>>> * issuer:
>>> CN=FQDN_OF_My_gateway_HOST,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
>>> * Server auth using Basic with user 'knox'
>>> > GET /gateway/default/webhdfs/v1/?op=LISTSTATUS HTTP/1.1
>>> > Authorization: Basic a25veDojMTIzUGFzc3dvcmQ=
>>> > User-Agent: curl/7.29.0
>>> > Host: localhost:8443
>>> > Accept: */*
>>> >
>>> < HTTP/1.1 401 Unauthorized
>>> HTTP/1.1 401 Unauthorized
>>> * Authentication problem. Ignoring this.
>>> < WWW-Authenticate: BASIC realm="application"
>>> WWW-Authenticate: BASIC realm="application"
>>> < Content-Length: 0
>>> Content-Length: 0
>>> < Server: Jetty(8.1.14.v20131031)
>>> Server: Jetty(8.1.14.v20131031)
>>>
>>>
>>> Please let me know if any additional information is required.
>>>
>>> Thanks,
>>> DP
>>>
>>>
>>
>
