Thanks Vipin for your analysis and support Regards, Prakash R
On Friday, January 5, 2018, Vipin Rathor <[email protected]> wrote: > So I build metron-rest jar and found that it does contain multiple > httpclient classes : > > $ jar -tf target/metron-rest-0.4.1.jar | grep -i "HttpClient.class" > org/apache/http/client/HttpClient.class > org/apache/hadoop/hbase/shaded/org/apache/http/client/HttpClient.class > org/apache/commons/httpclient/HttpClient.class > > The first one is indeed a HttpClient v4.5.2 which is causing problem in > your case. > > From the IntelliJ generated metron-rest.iml file, > $ grep httpclient metron-rest.iml > <orderEntry type="library" name="Maven: > org.apache.httpcomponents:httpclient:4.5.2" > level="project" /> > <orderEntry type="library" name="Maven: > commons-httpclient:commons-httpclient:3.1" > level="project" /> > > Interestingly, IntelliJ also reports that this library is not used at all > in metron-rest project and hence can be removed. > Since HttpClient v4.5.2 is known to cause trouble with Kerberos, we should > either remove it or downgrade it. > > Thinking of opening a Metron bug. > > @Simon/James, suggestions? > > > On Wed, Jan 3, 2018 at 6:33 PM, prakash r <[email protected]> wrote: > >> Hello Vipin >> >> I can see HttpClient related classes are loaded from metron-rest jar >> >> [Loaded org.apache.http.client.HttpClient from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.impl.client.HttpClientBuilder from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.conn.HttpClientConnectionManager from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.impl.client.CloseableHttpClient from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.impl.client.InternalHttpClient from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.impl.conn.PoolingHttpClientConnectionManager >> from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0. >> 0-51.jar] >> [Loaded org.apache.http.conn.HttpClientConnectionOperator from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.impl.conn.DefaultHttpClientConnectionOperator >> from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0. >> 0-51.jar] >> [Loaded >> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$ConfigData >> from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0. >> 0-51.jar] >> [Loaded >> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory >> from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0. >> 0-51.jar] >> [Loaded org.apache.http.impl.conn.ManagedHttpClientConnectionFactory >> from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0. >> 0-51.jar] >> [Loaded org.apache.http.HttpClientConnection from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.conn.ManagedHttpClientConnection from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.impl.DefaultBHttpClientConnection from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.impl.conn.DefaultManagedHttpClientConnection >> from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0. >> 0-51.jar] >> [Loaded org.apache.http.impl.client.HttpClientBuilder$2 from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.impl.client.AbstractHttpClient from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.springframework.web.client.HttpClientErrorException from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> [Loaded org.apache.http.client.protocol.HttpClientContext from >> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar] >> >> >> Regards, >> prakash R >> >> On Thu, Jan 4, 2018 at 1:26 PM, Vipin Rathor <[email protected]> wrote: >> >>> FYI, you can also try "-verbose:class" JVM command line option to check >>> what/where Metron REST daemon is loading. >>> >>> On Wed, Jan 3, 2018 at 6:24 PM, Vipin Rathor <[email protected]> wrote: >>> >>>> +user@metron >>>> >>>> Prakash, >>>> A quick look into Metron code tells me that Metron 0.4.1.1 (as well as >>>> HCP 1.3.0) was compiled with HttpClient v4.3.2. So this problem should not >>>> be there to begin with and every user would be complaining by now. >>>> If it is happening in your environment, this means that somehow a newer >>>> version of this library is being included. I'd start checking the classpath >>>> at this point and hopefully isolate & remove the problematic library. >>>> >>>> Hope this helps, >>>> VR >>>> >>>> On Wed, Jan 3, 2018 at 6:00 PM, prakash r <[email protected]> >>>> wrote: >>>> >>>>> Sorry still issue exists, im unable to degrade the httpclient alone >>>>> checking on the same. >>>>> is there any simple way where we can change the jar alone >>>>> >>>>> Regards, >>>>> prakash R >>>>> >>>>> On Thu, Jan 4, 2018 at 12:56 PM, prakash r <[email protected]> >>>>> wrote: >>>>> >>>>>> Thank you so much Vipin, >>>>>> Issue resolved by degrading httpclient >>>>>> >>>>>> >>>>>> Regards, >>>>>> Prakash R >>>>>> >>>>>> On Thu, Jan 4, 2018 at 11:14 AM, Vipin Rathor <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hello Prakash, >>>>>>> >>>>>>> Metron REST (or any Hadoop service for that matter) should not use >>>>>>> HTTPS/<host-fqdn>@<REALM> principal. If it is using this (as seen in >>>>>>> your >>>>>>> logs), most probably that is due to httpclient v4.5.2 library being >>>>>>> used by >>>>>>> Metron REST. This was a known issue and we have seen in past with Knox >>>>>>> etc. >>>>>>> [Reference: https://issues.apache.org/jira/browse/KNOX-762 ] >>>>>>> As a workaround/fix, please see if you can downgrade httpclient >>>>>>> library to v4.5.1. >>>>>>> >>>>>>> Thanks, >>>>>>> VR >>>>>>> >>>>>>> On Wed, Jan 3, 2018 at 3:54 PM, prakash r <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> HCP : 1.3.0 / Metron : 0.4.1.1 >>>>>>>> >>>>>>>> HDP : 2.5.0 >>>>>>>> >>>>>>>> >>>>>>>> Kerberos Authentication enabled for Hadoop cluster. >>>>>>>> >>>>>>>> When Metron Rest trying to connect to Storm, error is thrown as no >>>>>>>> Server not found in Kerberos database (7) - LOOKING_UP_SERVER >>>>>>>> >>>>>>>> >>>KRBError: cTime is Thu Oct 28 12:56:54 AEST 1971 57466614000 >>>>>>>> sTime is Wed Jan 03 22:57:12 AEDT 2018 1514980632000 suSec is 418131 >>>>>>>> error >>>>>>>> code is 7 error Message is Server not found in Kerberos database cname >>>>>>>> is >>>>>>>> [email protected] sname is >>>>>>>> *HTTPS/[email protected] >>>>>>>> <HTTPS/[email protected]> *msgType >>>>>>>> is 30 KrbException: Server not found in Kerberos database (7) - >>>>>>>> LOOKING_UP_SERVER at >>>>>>>> sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) >>>>>>>> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251) at >>>>>>>> sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262) at >>>>>>>> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308) >>>>>>>> at >>>>>>>> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126) >>>>>>>> at >>>>>>>> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) >>>>>>>> at >>>>>>>> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) >>>>>>>> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextIm >>>>>>>> pl.java:248) >>>>>>>> >>>>>>>> In KDC there is no principal with HTTPS/cbro-test-ms5.netwo >>>>>>>> [email protected] >>>>>>>> >>>>>>>> We can see only >>>>>>>> *HTTP/[email protected] >>>>>>>> <HTTP/[email protected]>* >>>>>>>> >>>>>>>> If we add manually principal (HTTPS/cbro-test-ms5.networks. >>>>>>>> [email protected]) using kadmin in kerberos server, >>>>>>>> getting error as checksum failed >>>>>>>> >>>>>>>> Jan 03, 2018 10:32:20 PM org.apache.catalina.core.StandardWrapperValve >>>>>>>> invoke SEVERE: Servlet.service() for servlet [dispatcherServlet] in >>>>>>>> context >>>>>>>> with path [] threw exception [Request processing failed; nested >>>>>>>> exception >>>>>>>> is org.springframework.web.client.RestClientException: Error >>>>>>>> running rest call; nested exception is >>>>>>>> org.springframework.web.client.HttpClientErrorException: >>>>>>>> 403 GSSException: Failure unspecified at GSS-API level (Mechanism >>>>>>>> level: >>>>>>>> Checksum failed)] with root cause >>>>>>>> org.springframework.web.client.HttpClientErrorException: >>>>>>>> 403 GSSException: Failure unspecified at GSS-API level (Mechanism >>>>>>>> level: >>>>>>>> Checksum failed) at org.springframework.web.client >>>>>>>> .DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) >>>>>>>> at >>>>>>>> org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) >>>>>>>> at org.springframework.web.client.RestTemplate.doExecute(RestTe >>>>>>>> mplate.java:620) >>>>>>>> >>>>>>>> Please suggest how to resolve this issue, thanks >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Prakash R >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -Rathor >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> -Rathor >>>> >>> >>> >>> >>> -- >>> -Rathor >>> >> >> > > > -- > -Rathor >
