Thanks Matt for a very detailed explanation, really appreciate that ! So yeah, I did start by searching HttpClient in all pom.xml. But that "mvn dependency:tree" was very important bit. In the interest of "not breaking anything major", I'd like to keep HttpClient around but at version 4.5.1. I'll do these changes in my sandbox first and see how does it look.
Thank you once again. Best Regards, VR On Thu, Jan 4, 2018 at 10:57 PM, Matt Foley <[email protected]> wrote: > This change will be a little tricky, because the problem is in _*indirect*_ > dependencies. In case you’re not a maven expert, here are some more > detailed instructions on how to do this. > > > > If you are correct that only the 4.5.2 version is causing you problems, > there are 3 instances, as seen in `mvn dependency:tree`: > > > > metron-rest > > org.springframework.security.kerberos:spring-security- > kerberos-client:jar:1.0.1.RELEASE:compile > > org.apache.httpcomponents:httpclient:jar:4.5.2:compile > > metron-enrichment > > com.maxmind.geoip2:geoip2:jar:2.8.0:compile > > org.apache.httpcomponents:httpclient:jar:4.5.2:compile > > metron-elasticsearch > > metron-enrichment > > com.maxmind.geoip2:geoip2:jar:2.8.0:compile > > org.apache.httpcomponents:httpclient:jar:4.5.2:compile > > > > It is not at all clear whether spring-security-kerberos-client and geoip2 > will be agreeable with downgrading httpclient, but you can try. > Downgrading only to 4.5.1 makes it more likely to succeed than if you > needed a larger downgrade. > > > > Because these are indirect dependencies, you must use the following > construct in the pom.xml: > > > > In metron-interface/metron-rest/pom.xml, WITHIN the declared dependency > on org.springframework.security.kerberos:spring-security- > kerberos-client:jar:1.0.1.RELEASE, add > > <exclusions> > > > <exclusion> > > <groupId>org.apache.httpcomponents</groupId> > > <artifactId>httpclient</artifactId> > > </exclusion> > > </exclusions> > > Then, at the SAME level as the dependency on spring-security-kerberos-client, > add an additional dependency as > > <dependency> > > <groupId>org.apache.httpcomponents</groupId> > > <artifactId>httpclient</artifactId> > > <version>4.5.1</version> > > </dependency> > > > > In metron-platform/metron-enrichment/pom.xml, WITHIN the declared > dependency on com.maxmind.geoip2:geoip2:jar:2.8.0, there are already > exclusions, so just add one more that says > > <exclusion> > > <groupId>org.apache.httpcomponents</groupId> > > <artifactId>httpclient</artifactId> > > </exclusion> > > Then at the level of the dependency, add as before > > <dependency> > > <groupId>org.apache.httpcomponents</groupId> > > <artifactId>httpclient</artifactId> > > <version>4.5.1</version> > > </dependency> > > > > You don’t need to change the pom for metron-elasticsearch, it will inherit > the change from metron-enrichment. > > > > Note: If you trust IntelliJ that the httpclient really isn’t needed, you > can insert the exclusions but leave out the new dependency declarations. I > have no opinion as to whether that will work with these two packages > (spring-security-kerberos-client and geoip2), but if you try leaving out > the 4.5.1 dependency, then I would be sure to carefully test the > functionality after the change. > > > > -------------------- > > The remainder of this message is general info about how to examine this > category of problem: > > > > Two useful commands are (executed from the root of a Metron code tree): > > mvn dependency:tree > deps.txt > > and > > grep -r --include pom.xml -i HttpClient * > > or > > grep -r --include pom.xml -B 3 -A 3 -i HttpClient *|more > > > > The grep will show you direct dependencies, and exclusions. > > The mvn dependency:tree command prints out all direct and (non-excluded) > indirect dependencies. > > The dependency tree was the starting point for my analysis above. > > > > Hope this helps, > > --Matt > > > > *From: *Vipin Rathor <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Thursday, January 4, 2018 at 8:02 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Metron Rest with Kerberos support > > > > So I build metron-rest jar and found that it does contain multiple > httpclient classes : > > > > $ jar -tf target/metron-rest-0.4.1.jar | grep -i "HttpClient.class" > > org/apache/http/client/HttpClient.class > > org/apache/hadoop/hbase/shaded/org/apache/http/client/HttpClient.class > > org/apache/commons/httpclient/HttpClient.class > > > > The first one is indeed a HttpClient v4.5.2 which is causing problem in > your case. > > > > From the IntelliJ generated metron-rest.iml file, > > $ grep httpclient metron-rest.iml > > <orderEntry type="library" name="Maven: > org.apache.httpcomponents:httpclient:4.5.2" > level="project" /> > > <orderEntry type="library" name="Maven: > commons-httpclient:commons-httpclient:3.1" > level="project" /> > > > > Interestingly, IntelliJ also reports that this library is not used at all > in metron-rest project and hence can be removed. > > Since HttpClient v4.5.2 is known to cause trouble with Kerberos, we should > either remove it or downgrade it. > > > > Thinking of opening a Metron bug. > > > > @Simon/James, suggestions? > > > > > > On Wed, Jan 3, 2018 at 6:33 PM, prakash r <[email protected]> wrote: > > Hello Vipin > > > > I can see HttpClient related classes are loaded from metron-rest jar > > > > [Loaded org.apache.http.client.HttpClient from file:/usr/hcp/1.3.0.0-51/ > metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.client.HttpClientBuilder from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.conn.HttpClientConnectionManager from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.client.CloseableHttpClient from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.client.InternalHttpClient from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.conn.PoolingHttpClientConnectionManager from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.conn.HttpClientConnectionOperator from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.conn.DefaultHttpClientConnectionOperator > from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded > org.apache.http.impl.conn.PoolingHttpClientConnectionManager$ConfigData > from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded > org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory > from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.conn.ManagedHttpClientConnectionFactory from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.HttpClientConnection from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.conn.ManagedHttpClientConnection from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.DefaultBHttpClientConnection from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.conn.DefaultManagedHttpClientConnection from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.client.HttpClientBuilder$2 from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.impl.client.AbstractHttpClient from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.springframework.web.client.HttpClientErrorException from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > [Loaded org.apache.http.client.protocol.HttpClientContext from > file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] > > > > > > Regards, > > prakash R > > > > On Thu, Jan 4, 2018 at 1:26 PM, Vipin Rathor <[email protected]> wrote: > > FYI, you can also try "-verbose:class" JVM command line option to check > what/where Metron REST daemon is loading. > > > > On Wed, Jan 3, 2018 at 6:24 PM, Vipin Rathor <[email protected]> wrote: > > +user@metron > > > > Prakash, > > A quick look into Metron code tells me that Metron 0.4.1.1 (as well as HCP > 1.3.0) was compiled with HttpClient v4.3.2. So this problem should not be > there to begin with and every user would be complaining by now. > > If it is happening in your environment, this means that somehow a newer > version of this library is being included. I'd start checking the classpath > at this point and hopefully isolate & remove the problematic library. > > > > Hope this helps, > > VR > > > > On Wed, Jan 3, 2018 at 6:00 PM, prakash r <[email protected]> wrote: > > Sorry still issue exists, im unable to degrade the httpclient alone > checking on the same. > > is there any simple way where we can change the jar alone > > > > Regards, > > prakash R > > > > On Thu, Jan 4, 2018 at 12:56 PM, prakash r <[email protected]> wrote: > > Thank you so much Vipin, > > Issue resolved by degrading httpclient > > > > > > Regards, > > Prakash R > > > > On Thu, Jan 4, 2018 at 11:14 AM, Vipin Rathor <[email protected]> wrote: > > Hello Prakash, > > > > Metron REST (or any Hadoop service for that matter) should not use > HTTPS/<host-fqdn>@<REALM> principal. If it is using this (as seen in your > logs), most probably that is due to httpclient v4.5.2 library being used by > Metron REST. This was a known issue and we have seen in past with Knox etc. > [Reference: https://issues.apache.org/jira/browse/KNOX-762 ] > > As a workaround/fix, please see if you can downgrade httpclient library to > v4.5.1. > > > > Thanks, > > VR > > > > On Wed, Jan 3, 2018 at 3:54 PM, prakash r <[email protected]> wrote: > > Hi, > > > > > > HCP : 1.3.0 / Metron : 0.4.1.1 > > > > HDP : 2.5.0 > > > > Kerberos Authentication enabled for Hadoop cluster. > > When Metron Rest trying to connect to Storm, error is thrown as no Server > not found in Kerberos database (7) - LOOKING_UP_SERVER > > >>>KRBError: cTime is Thu Oct 28 12:56:54 AEST 1971 57466614000 sTime is > Wed Jan 03 22:57:12 AEDT 2018 1514980632000 suSec is 418131 error code is 7 > error Message is Server not found in Kerberos database cname is > [email protected] sname is > *HTTPS/[email protected] > <HTTPS/[email protected]> *msgType is 30 > KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER > at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) at > sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251) at > sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262) at > sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308) > at > sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126) > at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) > at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) > at sun.security.jgss.GSSContextImpl.initSecContext( > GSSContextImpl.java:248) > > In KDC there is no principal with HTTPS/cbro-test-ms5. > [email protected] > > We can see only *HTTP/[email protected] > <HTTP/[email protected]>* > > If we add manually principal (HTTPS/cbro-test-ms5.networks. > [email protected]) using kadmin in kerberos server, getting error > as checksum failed > > Jan 03, 2018 10:32:20 PM org.apache.catalina.core.StandardWrapperValve > invoke SEVERE: Servlet.service() for servlet [dispatcherServlet] in context > with path [] threw exception [Request processing failed; nested exception > is org.springframework.web.client.RestClientException: Error running rest > call; nested exception is > org.springframework.web.client.HttpClientErrorException: > 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: > Checksum failed)] with root cause > org.springframework.web.client.HttpClientErrorException: > 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: > Checksum failed) at org.springframework.web.client. > DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) > at org.springframework.web.client.RestTemplate. > handleResponse(RestTemplate.java:667) at org.springframework.web. > client.RestTemplate.doExecute(RestTemplate.java:620) > > Please suggest how to resolve this issue, thanks > > > > > > Regards, > > Prakash R > > > > > > -- > > -Rathor > > > > > > > > > > -- > > -Rathor > > > > > > -- > > -Rathor > > > > > > > > -- > > -Rathor > -- -Rathor
