Dear Folks,
Anyone using REST client operations in Kerberos secure cluster.
We are facing multiple issues while preforming RemoteAdmin/RemoteHTable (REST
client) operations in a Kerberos secured cluster.
Issue - 1:
Client received below exception,
java.lang.IllegalStateException: Invalid use of BasicClientConnManager:
connection still allocated.
Make sure to release the connection before allocating another one.
at org.apache.http.util.Asserts.check(Asserts.java:34)
at
org.apache.http.impl.conn.BasicClientConnectionManager.getConnection(BasicClientConnectionManager.java:163)
at
org.apache.http.impl.conn.BasicClientConnectionManager$1.getConnection(BasicClientConnectionManager.java:145)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.hadoop.hbase.rest.client.Client.executeURI(Client.java:240)
at org.apache.hadoop.hbase.rest.client.Client.executePathOnly(Client.java:204)
at org.apache.hadoop.hbase.rest.client.Client.execute(Client.java:265)
at org.apache.hadoop.hbase.rest.client.Client.get(Client.java:459)
at org.apache.hadoop.hbase.rest.client.Client.get(Client.java:391)
at org.apache.hadoop.hbase.rest.client.Client.get(Client.java:376)
at
org.apache.hadoop.hbase.rest.client.RemoteAdmin.isTableAvailable(RemoteAdmin.java:254)
at
org.apache.hadoop.hbase.rest.client.RemoteAdmin.isTableAvailable(RemoteAdmin.java:99)
Soln:
Need to consume the response before negotiate at Client.executeURI(..)
https://github.com/apache/hbase/blob/3180a6864aa6a019fc1ec21ae78f009475bacea1/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java#L236
{code}
EntityUtils.consumeQuietly(resp.getEntity());
{code}
Issue - 2:
ServerHolder is not initialized with RESTServletContainer,
https://github.com/apache/hbase/blob/3180a6864aa6a019fc1ec21ae78f009475bacea1/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L248
Soln:
ServletHolder should be initialized as,
{code}
ServletHolder sh = new ServletHolder(new
RESTServletContainer(application));
{code}
Issue - 3:
GET (ExistsResource, ) request is working fine but none of the PUT/POST
request are working. Client is getting below exception always,
{noformat}
Exception in thread "main" java.security.PrivilegedActionException:
org.apache.http.NoHttpResponseException: host-name:port failed to respond
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.hbase.rest.RESTDemoClient.main(RESTDemoClient.java:80)
Caused by: org.apache.http.NoHttpResponseException: host-name:port failed to
respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:257)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:207)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:684)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
org.apache.hadoop.hbase.rest.client.Client.executeURI(Client.java:257)
at
org.apache.hadoop.hbase.rest.client.Client.executePathOnly(Client.java:216)
at org.apache.hadoop.hbase.rest.client.Client.execute(Client.java:282)
at org.apache.hadoop.hbase.rest.client.Client.put(Client.java:574)
at org.apache.hadoop.hbase.rest.client.Client.put(Client.java:521)
at org.apache.hadoop.hbase.rest.client.Client.put(Client.java:491)
at
org.apache.hadoop.hbase.rest.client.RemoteAdmin.createTable(RemoteAdmin.java:294)
However I am debugging the code, after negotiation client request never reached
to REST server side.
Anyone faced Issue-3, any pointer will be very much appreciated.
I will raise JIRA to address Issue 1 & 2 soon.
Thanks & Regards,
Pankaj
