So I build metron-rest jar and found that it does contain multiple
httpclient classes :
$ jar -tf target/metron-rest-0.4.1.jar | grep -i "HttpClient.class"
org/apache/http/client/HttpClient.class
org/apache/hadoop/hbase/shaded/org/apache/http/client/HttpClient.class
org/apache/commons/httpclient/HttpClient.class
The first one is indeed a HttpClient v4.5.2 which is causing problem in
your case.
>From the IntelliJ generated metron-rest.iml file,
$ grep httpclient metron-rest.iml
<orderEntry type="library" name="Maven:
org.apache.httpcomponents:httpclient:4.5.2" level="project" />
<orderEntry type="library" name="Maven:
commons-httpclient:commons-httpclient:3.1" level="project" />
Interestingly, IntelliJ also reports that this library is not used at all
in metron-rest project and hence can be removed.
Since HttpClient v4.5.2 is known to cause trouble with Kerberos, we should
either remove it or downgrade it.
Thinking of opening a Metron bug.
@Simon/James, suggestions?
On Wed, Jan 3, 2018 at 6:33 PM, prakash r <[email protected]> wrote:
> Hello Vipin
>
> I can see HttpClient related classes are loaded from metron-rest jar
>
> [Loaded org.apache.http.client.HttpClient from file:/usr/hcp/1.3.0.0-51/
> metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.client.HttpClientBuilder from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.conn.HttpClientConnectionManager from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.client.CloseableHttpClient from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.client.InternalHttpClient from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.conn.PoolingHttpClientConnectionManager from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.conn.HttpClientConnectionOperator from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.conn.DefaultHttpClientConnectionOperator
> from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$ConfigData
> from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory
> from file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.conn.ManagedHttpClientConnectionFactory from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.HttpClientConnection from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.conn.ManagedHttpClientConnection from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.DefaultBHttpClientConnection from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.conn.DefaultManagedHttpClientConnection from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.client.HttpClientBuilder$2 from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.impl.client.AbstractHttpClient from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.springframework.web.client.HttpClientErrorException from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
> [Loaded org.apache.http.client.protocol.HttpClientContext from
> file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]
>
>
> Regards,
> prakash R
>
> On Thu, Jan 4, 2018 at 1:26 PM, Vipin Rathor <[email protected]> wrote:
>
>> FYI, you can also try "-verbose:class" JVM command line option to check
>> what/where Metron REST daemon is loading.
>>
>> On Wed, Jan 3, 2018 at 6:24 PM, Vipin Rathor <[email protected]> wrote:
>>
>>> +user@metron
>>>
>>> Prakash,
>>> A quick look into Metron code tells me that Metron 0.4.1.1 (as well as
>>> HCP 1.3.0) was compiled with HttpClient v4.3.2. So this problem should not
>>> be there to begin with and every user would be complaining by now.
>>> If it is happening in your environment, this means that somehow a newer
>>> version of this library is being included. I'd start checking the classpath
>>> at this point and hopefully isolate & remove the problematic library.
>>>
>>> Hope this helps,
>>> VR
>>>
>>> On Wed, Jan 3, 2018 at 6:00 PM, prakash r <[email protected]>
>>> wrote:
>>>
>>>> Sorry still issue exists, im unable to degrade the httpclient alone
>>>> checking on the same.
>>>> is there any simple way where we can change the jar alone
>>>>
>>>> Regards,
>>>> prakash R
>>>>
>>>> On Thu, Jan 4, 2018 at 12:56 PM, prakash r <[email protected]>
>>>> wrote:
>>>>
>>>>> Thank you so much Vipin,
>>>>> Issue resolved by degrading httpclient
>>>>>
>>>>>
>>>>> Regards,
>>>>> Prakash R
>>>>>
>>>>> On Thu, Jan 4, 2018 at 11:14 AM, Vipin Rathor <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hello Prakash,
>>>>>>
>>>>>> Metron REST (or any Hadoop service for that matter) should not use
>>>>>> HTTPS/<host-fqdn>@<REALM> principal. If it is using this (as seen in your
>>>>>> logs), most probably that is due to httpclient v4.5.2 library being used
>>>>>> by
>>>>>> Metron REST. This was a known issue and we have seen in past with Knox
>>>>>> etc.
>>>>>> [Reference: https://issues.apache.org/jira/browse/KNOX-762 ]
>>>>>> As a workaround/fix, please see if you can downgrade httpclient
>>>>>> library to v4.5.1.
>>>>>>
>>>>>> Thanks,
>>>>>> VR
>>>>>>
>>>>>> On Wed, Jan 3, 2018 at 3:54 PM, prakash r <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>> HCP : 1.3.0 / Metron : 0.4.1.1
>>>>>>>
>>>>>>> HDP : 2.5.0
>>>>>>>
>>>>>>>
>>>>>>> Kerberos Authentication enabled for Hadoop cluster.
>>>>>>>
>>>>>>> When Metron Rest trying to connect to Storm, error is thrown as no
>>>>>>> Server not found in Kerberos database (7) - LOOKING_UP_SERVER
>>>>>>>
>>>>>>> >>>KRBError: cTime is Thu Oct 28 12:56:54 AEST 1971 57466614000
>>>>>>> sTime is Wed Jan 03 22:57:12 AEDT 2018 1514980632000 suSec is 418131
>>>>>>> error
>>>>>>> code is 7 error Message is Server not found in Kerberos database cname
>>>>>>> is
>>>>>>> [email protected] sname is
>>>>>>> *HTTPS/[email protected]
>>>>>>> <HTTPS/[email protected]> *msgType
>>>>>>> is 30 KrbException: Server not found in Kerberos database (7) -
>>>>>>> LOOKING_UP_SERVER at
>>>>>>> sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
>>>>>>> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251) at
>>>>>>> sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262) at
>>>>>>> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
>>>>>>> at
>>>>>>> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
>>>>>>> at
>>>>>>> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
>>>>>>> at
>>>>>>> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
>>>>>>> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextIm
>>>>>>> pl.java:248)
>>>>>>>
>>>>>>> In KDC there is no principal with HTTPS/cbro-test-ms5.netwo
>>>>>>> [email protected]
>>>>>>>
>>>>>>> We can see only
>>>>>>> *HTTP/[email protected]
>>>>>>> <HTTP/[email protected]>*
>>>>>>>
>>>>>>> If we add manually principal (HTTPS/cbro-test-ms5.networks.
>>>>>>> [email protected]) using kadmin in kerberos server, getting
>>>>>>> error as checksum failed
>>>>>>>
>>>>>>> Jan 03, 2018 10:32:20 PM org.apache.catalina.core.StandardWrapperValve
>>>>>>> invoke SEVERE: Servlet.service() for servlet [dispatcherServlet] in
>>>>>>> context
>>>>>>> with path [] threw exception [Request processing failed; nested
>>>>>>> exception
>>>>>>> is org.springframework.web.client.RestClientException: Error
>>>>>>> running rest call; nested exception is
>>>>>>> org.springframework.web.client.HttpClientErrorException:
>>>>>>> 403 GSSException: Failure unspecified at GSS-API level (Mechanism level:
>>>>>>> Checksum failed)] with root cause
>>>>>>> org.springframework.web.client.HttpClientErrorException:
>>>>>>> 403 GSSException: Failure unspecified at GSS-API level (Mechanism level:
>>>>>>> Checksum failed) at org.springframework.web.client
>>>>>>> .DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
>>>>>>> at
>>>>>>> org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667)
>>>>>>> at org.springframework.web.client.RestTemplate.doExecute(RestTe
>>>>>>> mplate.java:620)
>>>>>>>
>>>>>>> Please suggest how to resolve this issue, thanks
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Prakash R
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Rathor
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> -Rathor
>>>
>>
>>
>>
>> --
>> -Rathor
>>
>
>
--
-Rathor
