Hello Vipin I can see HttpClient related classes are loaded from metron-rest jar
[Loaded org.apache.http.client.HttpClient from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.HttpClientBuilder from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.conn.HttpClientConnectionManager from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.CloseableHttpClient from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.InternalHttpClient from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.PoolingHttpClientConnectionManager from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.conn.HttpClientConnectionOperator from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.DefaultHttpClientConnectionOperator from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.PoolingHttpClientConnectionManager$ConfigData from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.ManagedHttpClientConnectionFactory from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.HttpClientConnection from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.conn.ManagedHttpClientConnection from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.DefaultBHttpClientConnection from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.conn.DefaultManagedHttpClientConnection from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.HttpClientBuilder$2 from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.impl.client.AbstractHttpClient from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.springframework.web.client.HttpClientErrorException from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] [Loaded org.apache.http.client.protocol.HttpClientContext from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] Regards, prakash R On Thu, Jan 4, 2018 at 1:26 PM, Vipin Rathor <[email protected]> wrote: > FYI, you can also try "-verbose:class" JVM command line option to check > what/where Metron REST daemon is loading. > > On Wed, Jan 3, 2018 at 6:24 PM, Vipin Rathor <[email protected]> wrote: > >> +user@metron >> >> Prakash, >> A quick look into Metron code tells me that Metron 0.4.1.1 (as well as >> HCP 1.3.0) was compiled with HttpClient v4.3.2. So this problem should not >> be there to begin with and every user would be complaining by now. >> If it is happening in your environment, this means that somehow a newer >> version of this library is being included. I'd start checking the classpath >> at this point and hopefully isolate & remove the problematic library. >> >> Hope this helps, >> VR >> >> On Wed, Jan 3, 2018 at 6:00 PM, prakash r <[email protected]> wrote: >> >>> Sorry still issue exists, im unable to degrade the httpclient alone >>> checking on the same. >>> is there any simple way where we can change the jar alone >>> >>> Regards, >>> prakash R >>> >>> On Thu, Jan 4, 2018 at 12:56 PM, prakash r <[email protected]> >>> wrote: >>> >>>> Thank you so much Vipin, >>>> Issue resolved by degrading httpclient >>>> >>>> >>>> Regards, >>>> Prakash R >>>> >>>> On Thu, Jan 4, 2018 at 11:14 AM, Vipin Rathor <[email protected]> >>>> wrote: >>>> >>>>> Hello Prakash, >>>>> >>>>> Metron REST (or any Hadoop service for that matter) should not use >>>>> HTTPS/<host-fqdn>@<REALM> principal. If it is using this (as seen in your >>>>> logs), most probably that is due to httpclient v4.5.2 library being used >>>>> by >>>>> Metron REST. This was a known issue and we have seen in past with Knox >>>>> etc. >>>>> [Reference: https://issues.apache.org/jira/browse/KNOX-762 ] >>>>> As a workaround/fix, please see if you can downgrade httpclient >>>>> library to v4.5.1. >>>>> >>>>> Thanks, >>>>> VR >>>>> >>>>> On Wed, Jan 3, 2018 at 3:54 PM, prakash r <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> HCP : 1.3.0 / Metron : 0.4.1.1 >>>>>> >>>>>> HDP : 2.5.0 >>>>>> >>>>>> >>>>>> Kerberos Authentication enabled for Hadoop cluster. >>>>>> >>>>>> When Metron Rest trying to connect to Storm, error is thrown as no >>>>>> Server not found in Kerberos database (7) - LOOKING_UP_SERVER >>>>>> >>>>>> >>>KRBError: cTime is Thu Oct 28 12:56:54 AEST 1971 57466614000 sTime >>>>>> is Wed Jan 03 22:57:12 AEDT 2018 1514980632000 suSec is 418131 error code >>>>>> is 7 error Message is Server not found in Kerberos database cname is >>>>>> [email protected] sname is >>>>>> *HTTPS/[email protected] >>>>>> <HTTPS/[email protected]> *msgType is >>>>>> 30 KrbException: Server not found in Kerberos database (7) - >>>>>> LOOKING_UP_SERVER at >>>>>> sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) >>>>>> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251) at >>>>>> sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262) at >>>>>> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308) >>>>>> at >>>>>> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126) >>>>>> at >>>>>> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) >>>>>> at >>>>>> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) >>>>>> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextIm >>>>>> pl.java:248) >>>>>> >>>>>> In KDC there is no principal with HTTPS/cbro-test-ms5.netwo >>>>>> [email protected] >>>>>> >>>>>> We can see only >>>>>> *HTTP/[email protected] >>>>>> <HTTP/[email protected]>* >>>>>> >>>>>> If we add manually principal (HTTPS/cbro-test-ms5.networks. >>>>>> [email protected]) using kadmin in kerberos server, getting >>>>>> error as checksum failed >>>>>> >>>>>> Jan 03, 2018 10:32:20 PM org.apache.catalina.core.StandardWrapperValve >>>>>> invoke SEVERE: Servlet.service() for servlet [dispatcherServlet] in >>>>>> context >>>>>> with path [] threw exception [Request processing failed; nested exception >>>>>> is org.springframework.web.client.RestClientException: Error running >>>>>> rest call; nested exception is >>>>>> org.springframework.web.client.HttpClientErrorException: >>>>>> 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: >>>>>> Checksum failed)] with root cause >>>>>> org.springframework.web.client.HttpClientErrorException: >>>>>> 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: >>>>>> Checksum failed) at org.springframework.web.client >>>>>> .DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) >>>>>> at >>>>>> org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) >>>>>> at org.springframework.web.client.RestTemplate.doExecute(RestTe >>>>>> mplate.java:620) >>>>>> >>>>>> Please suggest how to resolve this issue, thanks >>>>>> >>>>>> >>>>>> Regards, >>>>>> Prakash R >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> -Rathor >>>>> >>>> >>>> >>> >> >> >> -- >> -Rathor >> > > > > -- > -Rathor >
