Hi Larry, I can confirm, that exchanging the httpclient-JAR fixes the problem! Thanks for your help!
I tried with the httpclient-4.4.1.jar (I did not have the 4.5.1 JAR around) and the same cURL-Call that failed before now succeeds. For reference: I removed the file /usr/hdp/2.5.0.0-1245/knox/dep/httpclient-4.5.2.jar and copied the file /usr/hdp/2.5.0.0-1245/falcon/client/lib/httpclient-4.4.1.jar to the path /usr/hdp/2.5.0.0-1245/falcon/client/lib/ instead. After restarting Knox, I tested again. Thanks again, Benjamin Von: larry mccay [mailto:[email protected]] Gesendet: Freitag, 21. Oktober 2016 15:44 An: [email protected] Betreff: Re: WebHDFS over Knox 0.9.0 not working in secure HDP 2.5 cluster (Kerberos and SSL) You can try to workaround this by replacing the dep/httpclient-4.5.2.jar jar with the 4.5.1 version. 4.5.1 did not include the change that broke 4.5.2. Hopefully, there is no use of any incompatible changes in our dispatch classes. Please let me know if you can get by this error with this workaround. On Fri, Oct 21, 2016 at 8:38 AM, larry mccay <[email protected]<mailto:[email protected]>> wrote: From what I can see, this problem is directly related to: https://issues.apache.org/jira/browse/HTTPCLIENT-1712. I have asked them to provide a release that removes this incorrect patch but we will likely have to deal with it in Knox - if at all possible. I will look into overriding GGSSchemeBase in Knox and figure out how to use the extension or forked version as a downloadable patch. Sorry for the inconvenience! On Fri, Oct 21, 2016 at 7:48 AM, larry mccay <[email protected]<mailto:[email protected]>> wrote: Hi Benjamin - I suspect, based on the error message, that you are right. The HTTP service name in the SPN is incorrectly set as HTTPS. Not sure why this would be. I will look into our kerberos dispatch code and see if we are explicitly setting this for some reason. We should be just letting HttpClient do it for us but I will check. thanks, --larry On Fri, Oct 21, 2016 at 4:59 AM, Ruland, Benjamin <[email protected]<mailto:[email protected]>> wrote: Hi everyone, I am experiencing problems with Knox while using WebHDFS in a cluster with Kerberos and SSL. The KDC is an Microsoft AD 2012. Kerberos-Encryption is set to AES256. Knox is connected to AD via LDAP sync (this is working fine for other Knox services). I am running HDP 2.5 with Knox 0.9.0 In general, the cluster runs fine. WebHDFS using SPNEGO is working. But when accessing WebHDFS over Knox, I get an 401 error and some strange logs. I suspect that Knox is trying to get a ticket for a HTTPS/namenode@REALM principal, which does not exist. Although running SSL, all principals for SPNEGO are HTTP/... I this a Knox Bug or is this a misconfiguration at some point? It would be great, if someone has advice. Best regards, Benjamin The used command is: [root@utilitynode ~]# curl -ik -u validuser "https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS"; Enter host password for user 'validuser': HTTP/1.1 401 Unauthorized Date: Wed, 12 Oct 2016 07:47:41 GMT Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Tue,11-Oct-2016 07:47:41 GMT WWW-Authenticate: BASIC realm="application" Content-Length: 0 Server: Jetty(9.2.15.v20160210) Debug Log in knox gateway.log 2016-10-12 09:51:49,735 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /webhdfs/v1/ 2016-10-12 09:51:49,740 DEBUG hadoop.gateway (KnoxLdapRealm.java:getUserDn(673)) - Searching from OU=someOU,DC=somedomain,DC=de where (&(objectclass=person)(sAMAccountName=validuser)) scope subtree 2016-10-12 09:51:49,745 INFO hadoop.gateway (KnoxLdapRealm.java:getUserDn(679)) - Computed userDn: CN=validuser,OU=Users,OU=someOU,DC=somedomain,DC=de using ldapSearch for principal: validuser 2016-10-12 09:51:49,749 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: https://utilitynode:8443/gateway/default/webhdfs/v1/?OP=LISTSTATUS, direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/root to URL: https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS 2016-10-12 09:51:49,749 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(120)) - Dispatch request: GET https://utilitynode.somedomain.de:50470/webhdfs/v1/?OP=LISTSTATUS&doAs=validuser 2016-10-12 09:51:49,781 WARN auth.HttpAuthenticator (HttpAuthenticator.java:generateAuthResponse(207)) - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))) 2016-10-12 09:51:49,782 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(133)) - Dispatch response status: 401 2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit character set ISO-8859-1 for entity of type text/html 2016-10-12 09:51:49,783 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound response entity content type: text/html; charset=iso-8859-1 Log in knox gateway.out Found ticket for knox/[email protected]<mailto:[email protected]> to go to krbtgt/[email protected]<mailto:[email protected]> expiring on Wed Oct 12 19:53:51 CEST 2016 Entered Krb5Context.initSecContext with state=STATE_NEW Service ticket not found in the subject >>> Credentials acquireServiceCreds: same realm default etypes for default_tgs_enctypes: 18. >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType getKDCFromDNS using UDP >>> KrbKdcReq send: >>> kdc=domaincontroller.somedomain.de<http://domaincontroller.somedomain.de>. >>> TCP:88, timeout=30000, number of retries =3, #bytes=1661 >>> KDCCommunication: >>> kdc=domaincontroller.somedomain.de<http://domaincontroller.somedomain.de>. >>> TCP:88, timeout=30000,Attempt =1, #bytes=1661 >>>DEBUG: TCPClient reading 127 bytes >>> KrbKdcReq send: #bytes read=127 >>> KdcAccessibility: remove >>> domaincontroller.somedomain.de<http://domaincontroller.somedomain.de>.:88 >>> KDCRep: init() encoding tag is 126 req type is 13 >>>KRBError: sTime is Wed Oct 12 09:53:51 CEST 2016 1476258831000 suSec is 8354 suSec is 8354 error code is 7 error Message is Server not found in Kerberos database sname is HTTPS/[email protected]<mailto:[email protected]> msgType is 30 Extracts from topology config: <topology> <gateway> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <!-- LDAP Sync properties sit here --> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> <provider> <role>authorization</role> <name>XASecurePDPKnox</name> <enabled>true</enabled> </provider> <provider> <role>ha</role> <name>HaProvider</name> <enabled>true</enabled> <param> <name>WEBHDFS</name> <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value> </param> </provider> </gateway> <service> <role>NAMENODE</role> <url>hdfs://namenode.somedomain.de:8020<http://namenode.somedomain.de:8020></url> <url>hdfs://namenode2.somedomain.de:8020<http://namenode2.somedomain.de:8020></url> </service> <service> <role>WEBHDFS</role> <url>https://namenode.somedomain.de:50470/webhdfs</url> <url>https://namenode2.somedomain.de:50470/webhdfs</url> </service> </topology> ----------------------------------- Computacenter AG & Co. oHG, mit Sitz in Kerpen (Amtsgericht Köln HRA 18096) Vertretungsberechtigte Gesellschafter: Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384) Vorstand: Tony Conophy Aufsichtsrat: Michael Norris (Vorsitzender) Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284) Geschäftsführer: Dr. Karsten Freihube, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck Visit us on the Internet: http://www.computacenter.de<http://www.computacenter.de/> Visit our Online-Shop: https://shop.computacenter.de<https://shop.computacenter.de/> This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document. -----------------------------------
