This change will be a little tricky, because the problem is in _indirect_ dependencies. In case you’re not a maven expert, here are some more detailed instructions on how to do this.
If you are correct that only the 4.5.2 version is causing you problems, there are 3 instances, as seen in `mvn dependency:tree`: metron-rest org.springframework.security.kerberos:spring-security-kerberos-client:jar:1.0.1.RELEASE:compile org.apache.httpcomponents:httpclient:jar:4.5.2:compile metron-enrichment com.maxmind.geoip2:geoip2:jar:2.8.0:compile org.apache.httpcomponents:httpclient:jar:4.5.2:compile metron-elasticsearch metron-enrichment com.maxmind.geoip2:geoip2:jar:2.8.0:compile org.apache.httpcomponents:httpclient:jar:4.5.2:compile It is not at all clear whether spring-security-kerberos-client and geoip2 will be agreeable with downgrading httpclient, but you can try. Downgrading only to 4.5.1 makes it more likely to succeed than if you needed a larger downgrade. Because these are indirect dependencies, you must use the following construct in the pom.xml: In metron-interface/metron-rest/pom.xml, WITHIN the declared dependency on org.springframework.security.kerberos:spring-security-kerberos-client:jar:1.0.1.RELEASE, add <exclusions> <exclusion> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> </exclusion> </exclusions> Then, at the SAME level as the dependency on spring-security-kerberos-client, add an additional dependency as <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.1</version> </dependency> In metron-platform/metron-enrichment/pom.xml, WITHIN the declared dependency on com.maxmind.geoip2:geoip2:jar:2.8.0, there are already exclusions, so just add one more that says <exclusion> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> </exclusion> Then at the level of the dependency, add as before <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.1</version> </dependency> You don’t need to change the pom for metron-elasticsearch, it will inherit the change from metron-enrichment. Note: If you trust IntelliJ that the httpclient really isn’t needed, you can insert the exclusions but leave out the new dependency declarations. I have no opinion as to whether that will work with these two packages (spring-security-kerberos-client and geoip2), but if you try leaving out the 4.5.1 dependency, then I would be sure to carefully test the functionality after the change. -------------------- The remainder of this message is general info about how to examine this category of problem: Two useful commands are (executed from the root of a Metron code tree): mvn dependency:tree > deps.txt and grep -r --include pom.xml -i HttpClient * or grep -r --include pom.xml -B 3 -A 3 -i HttpClient *|more The grep will show you direct dependencies, and exclusions. The mvn dependency:tree command prints out all direct and (non-excluded) indirect dependencies. The dependency tree was the starting point for my analysis above. Hope this helps, --Matt From: Vipin Rathor <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, January 4, 2018 at 8:02 PM To: "[email protected]" <[email protected]> Subject: Re: Metron Rest with Kerberos support So I build metron-rest jar and found that it does contain multiple httpclient classes : $ jar -tf target/metron-rest-0.4.1.jar | grep -i "HttpClient.class" org/apache/http/client/HttpClient.class org/apache/hadoop/hbase/shaded/org/apache/http/client/HttpClient.class org/apache/commons/httpclient/HttpClient.class The first one is indeed a HttpClient v4.5.2 which is causing problem in your case. >From the IntelliJ generated metron-rest.iml file, $ grep httpclient metron-rest.iml <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.5.2" level="project" /> <orderEntry type="library" name="Maven: commons-httpclient:commons-httpclient:3.1" level="project" /> Interestingly, IntelliJ also reports that this library is not used at all in metron-rest project and hence can be removed. Since HttpClient v4.5.2 is known to cause trouble with Kerberos, we should either remove it or downgrade it. Thinking of opening a Metron bug. @Simon/James, suggestions? On Wed, Jan 3, 2018 at 6:33 PM, prakash r <[email protected]> wrote: Hello Vipin I can see HttpClient related classes are loaded from metron-rest jar [Loaded org.apache.http.client.HttpClient from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.HttpClientBuilder from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.conn.HttpClientConnectionManager from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.CloseableHttpClient from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.InternalHttpClient from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.PoolingHttpClientConnectionManager from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.conn.HttpClientConnectionOperator from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.DefaultHttpClientConnectionOperator from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.PoolingHttpClientConnectionManager$ConfigData from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.ManagedHttpClientConnectionFactory from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.HttpClientConnection from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.conn.ManagedHttpClientConnection from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.DefaultBHttpClientConnection from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.DefaultManagedHttpClientConnection from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.HttpClientBuilder$2 from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.AbstractHttpClient from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.springframework.web.client.HttpClientErrorException from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.client.protocol.HttpClientContext from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] Regards, prakash R On Thu, Jan 4, 2018 at 1:26 PM, Vipin Rathor <[email protected]> wrote: FYI, you can also try "-verbose:class" JVM command line option to check what/where Metron REST daemon is loading. On Wed, Jan 3, 2018 at 6:24 PM, Vipin Rathor <[email protected]> wrote: +user@metron Prakash, A quick look into Metron code tells me that Metron 0.4.1.1 (as well as HCP 1.3.0) was compiled with HttpClient v4.3.2. So this problem should not be there to begin with and every user would be complaining by now. If it is happening in your environment, this means that somehow a newer version of this library is being included. I'd start checking the classpath at this point and hopefully isolate & remove the problematic library. Hope this helps, VR On Wed, Jan 3, 2018 at 6:00 PM, prakash r <[email protected]> wrote: Sorry still issue exists, im unable to degrade the httpclient alone checking on the same. is there any simple way where we can change the jar alone Regards, prakash R On Thu, Jan 4, 2018 at 12:56 PM, prakash r <[email protected]> wrote: Thank you so much Vipin, Issue resolved by degrading httpclient Regards, Prakash R On Thu, Jan 4, 2018 at 11:14 AM, Vipin Rathor <[email protected]> wrote: Hello Prakash, Metron REST (or any Hadoop service for that matter) should not use HTTPS/<host-fqdn>@<REALM> principal. If it is using this (as seen in your logs), most probably that is due to httpclient v4.5.2 library being used by Metron REST. This was a known issue and we have seen in past with Knox etc. [Reference: https://issues.apache.org/jira/browse/KNOX-762 ] As a workaround/fix, please see if you can downgrade httpclient library to v4.5.1. Thanks, VR On Wed, Jan 3, 2018 at 3:54 PM, prakash r <[email protected]> wrote: Hi, HCP : 1.3.0 / Metron : 0.4.1.1 HDP : 2.5.0 Kerberos Authentication enabled for Hadoop cluster. When Metron Rest trying to connect to Storm, error is thrown as no Server not found in Kerberos database (7) - LOOKING_UP_SERVER >>>KRBError: cTime is Thu Oct 28 12:56:54 AEST 1971 57466614000 sTime is Wed >>>Jan 03 22:57:12 AEDT 2018 1514980632000 suSec is 418131 error code is 7 >>>error Message is Server not found in Kerberos database cname is >>>[email protected] sname is >>>HTTPS/[email protected] msgType is 30 >>>KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER >>>at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) at >>>sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251) at >>>sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262) at >>>sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308) >>> at >>>sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126) >>> at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) >>>at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) >>>at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) In KDC there is no principal with HTTPS/[email protected] We can see only HTTP/[email protected] If we add manually principal (HTTPS/[email protected]) using kadmin in kerberos server, getting error as checksum failed Jan 03, 2018 10:32:20 PM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)] with root cause org.springframework.web.client.HttpClientErrorException: 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) Please suggest how to resolve this issue, thanks Regards, Prakash R -- -Rathor -- -Rathor -- -Rathor -- -Rathor
