pkuwm opened a new issue #1471: URL: https://github.com/apache/helix/issues/1471
### Describe the bug We found that `ConnectionPoolTimeoutException` is thrown from `CustomRestClientImpl` and cause custom health check failure. ``` [INFO ] 2020-10-16 13:10:54,273 org.apache.helix.rest.client.CustomRestClient - Executing request: POST http://localhost:11933/instanceHealthStatus HTTP/1.1, headers: [Accept: application/json], entity: [Content-Type: application/json; charset=UTF-8,Content-Length: 2,Chunked: false] [ERROR] 2020-10-16 13:11:54,281 org.apache.helix.rest.client.CustomRestClient - Failed to perform customized health check. Is participant endpoint http://localhost:119 33/instanceHealthStatus available? org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at org.apache.helix.rest.client.CustomRestClientImpl.post(CustomRestClientImpl.java:140) at org.apache.helix.rest.client.CustomRestClientImpl.getInstanceStoppableCheck(CustomRestClientImpl.java:75) at org.apache.helix.rest.server.service.InstanceServiceImpl.performCustomInstanceCheck(InstanceServiceImpl.java:259) at org.apache.helix.rest.server.service.InstanceServiceImpl.lambda$null$2(InstanceServiceImpl.java:200) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` ### To Reproduce Change code a bit to let `handleResponse` always go to the else branch that does not consume entity and throw `ClientProtocolException`. ### Expected behavior Should not have `ConnectionPoolTimeoutException`. ### Additional context If custom health check rest calls return non-200 responses, the entity is not consumed so the connection is still occupied, which causes connection leakage. We need to consume the response entity before throwing an exception for non-200 responses, so the stream or connection is released. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
