Dear all, Using the Apache HttpAsyncClient inside Elasticsearch's RestHighLevelClient, I found a problem that HttpAsyncClient might throw TimeoutException for successful requests due to failed lease requests.
The problem is that AbstractNIOConnPool throws TimeoutException for a lease request inside release() method, indicating the HttpRequest has timed out, while it has actually succeeded and has nothing to do with the following lease request. Inside org.apache.http.nio.pool.AbstractNIOConnPool#release method, after released the connection to the pool, it goes to processNextPendingRequest(), which then calls processPendingRequest() for LeaseRequests. However, during processing LeaseRequests, it might throw TimeoutException due to exceeding the connectionRequestTimeout deadline. The TimeoutException is thrown up through the call stack, making release() failed and further making the related HttpResponse failed. Then the affected HttpResponse passes the TimeoutException to the user while this request/response has actually succeeded. To make things worse, together with some other problems, like improper config params, it might lead to potential memory leak due to unlimited queued lease requests. This is my detailed description in SO https://stackoverflow.com/questions/57552172/apache-httpasyncclient-all-connections-timed-out-suddenly-and-all-future-reque/57584279 Also, this problem has been discussed several times in the Elasticsearch community: https://github.com/elastic/elasticsearch/issues/24069 https://stackoverflow.com/a/48480305/5193455 https://hibernate.atlassian.net/browse/HSEARCH-2681 https://discuss.elastic.co/t/elasticsearch-in-docker-and-the-restclient-often-got-java-util-concurrent-timeoutexception/68118 The general solution is calling RequestConfig.Builder.setConnectionRequestTimeout(0). My suggestion is that perhaps the TimeoutException for lease requests should be thrown internally to not affect successful HttpResponses. Best regards, Zhao Bofeng
