On Wed, Feb 16, 2011 at 3:14 PM, Oleg Kalnichevski <[email protected]> wrote: > > Once aborted a request cannot be executed again. Create a new HttpGet > instance for each request execution. They are cheap. > > The problem is that HttpClient does not check whether or not a request > has been aborted prior to trying to execute it. Feel free to raise a > JIRA for this defect. > > Oleg >
I've implemented you comments and the execution has greatly improved. The code is working :) Please find code at https://gist.github.com/829061/50faa10a99aea3315449e13b4eaf558469795778 Several things still bug me: 1. Why (as you can see from the attached below execution log) the code runs cleanly from ~20seconds and only then starts very frequently to throw ConnectionPoolTimeoutException ? Is this a problem on my end, the network or the linux kernel. I would appreciate if someone could download the HttpClientTest.java and run it to see if this behaviour reproduces for him as well. In my setup I've ran this now for ~50 times and in all cases the errors start after ~20seconds. 2. Do I need to call httpGet.abort(); on the old object is I'm going to create a new one on the next loop? (Please look at the code, it much easier just to show the logic then try and explain it here in text). 3. Is the code written correctly to reuse the connection, I would like to verify I'm send the GET events as quickly as possible - I will always be talking to the same RoutePath and Hench would like to take advantage of connection pooling. (After I get the buggy code to work correctly I will obviously raise the MaxPerRoute and the CONNECTION_TIMEOUT to reasonably values. 4. You say: > Once aborted a request cannot be executed again. Create a new HttpGet > instance for each request execution. They are cheap. What is considered resource intensive to create instances of? Is new DefaultHttpClient(); costly? Is new ThreadSafeClientConnManager(); costly? 5. Please pay attention to my comment regarding the fact that my code is mostly based on the http client official documentation, I think it's worth updating it so that other people won't be confused. Would you like me to file a JIRA for it? Is there a code repository for the documentation, I would rather submit a patch. Thanks for the great help so far! Maxim. Execution log: Wed Feb 16 16:25:59 IST 2011: Starting thread execution Wed Feb 16 16:25:59 IST 2011 Thread-1: new HttpGet Wed Feb 16 16:25:59 IST 2011 Thread-4: new HttpGet Wed Feb 16 16:25:59 IST 2011 Thread-2: new HttpGet Wed Feb 16 16:25:59 IST 2011 Thread-3: new HttpGet Wed Feb 16 16:26:21 IST 2011 Thread-1: thread failure #1, System did a total of 10530 GET requests Wed Feb 16 16:26:21 IST 2011 Thread-1: Exception Wed Feb 16 16:26:21 IST 2011 Thread-1: Abort Wed Feb 16 16:26:21 IST 2011 Thread-1: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:21 IST 2011 Thread-1: new HttpGet org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) Wed Feb 16 16:26:24 IST 2011 Thread-2: thread failure #1, System did a total of 12253 GET requests org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) Wed Feb 16 16:26:24 IST 2011 Thread-2: Exception Wed Feb 16 16:26:24 IST 2011 Thread-2: Abort Wed Feb 16 16:26:24 IST 2011 Thread-2: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:24 IST 2011 Thread-2: new HttpGet at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) Wed Feb 16 16:26:25 IST 2011 Thread-1: thread failure #2, System did a total of 12977 GET requests org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) Wed Feb 16 16:26:25 IST 2011 Thread-1: Exception Wed Feb 16 16:26:25 IST 2011 Thread-1: Abort Wed Feb 16 16:26:25 IST 2011 Thread-1: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:25 IST 2011 Thread-1: new HttpGet Wed Feb 16 16:26:29 IST 2011 Thread-1: thread failure #3, System did a total of 15392 GET requests org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224)Wed Feb 16 16:26:29 IST 2011 Thread-1: Exception Wed Feb 16 16:26:29 IST 2011 Thread-1: Abort Wed Feb 16 16:26:29 IST 2011 Thread-1: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:29 IST 2011 Thread-1: new HttpGet at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) Wed Feb 16 16:26:30 IST 2011 Thread-2: thread failure #2, System did a total of 16072 GET requests org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connectionWed Feb 16 16:26:30 IST 2011 Thread-2: Exception Wed Feb 16 16:26:30 IST 2011 Thread-2: Abort Wed Feb 16 16:26:30 IST 2011 Thread-2: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:30 IST 2011 Thread-2: new HttpGet at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) Wed Feb 16 16:26:31 IST 2011 Thread-4: thread failure #1, System did a total of 16173 GET requests Wed Feb 16 16:26:31 IST 2011 Thread-4: Exception Wed Feb 16 16:26:31 IST 2011 Thread-4: Abort Wed Feb 16 16:26:31 IST 2011 Thread-4: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:31 IST 2011 Thread-4: new HttpGet Wed Feb 16 16:26:32 IST 2011 Thread-3: thread failure #1, System did a total of 16726 GET requests org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) Wed Feb 16 16:26:32 IST 2011 Thread-3: Exception Wed Feb 16 16:26:32 IST 2011 Thread-3: Abort Wed Feb 16 16:26:32 IST 2011 Thread-3: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:32 IST 2011 Thread-3: new HttpGet at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) Wed Feb 16 16:26:33 IST 2011 Thread-3: thread failure #2, System did a total of 17635 GET requests org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) Wed Feb 16 16:26:33 IST 2011 Thread-3: Exception Wed Feb 16 16:26:33 IST 2011 Thread-3: Abort Wed Feb 16 16:26:33 IST 2011 Thread-3: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:33 IST 2011 Thread-3: new HttpGet Wed Feb 16 16:26:33 IST 2011 Thread-4: thread failure #2, System did a total of 17659 GET requests Wed Feb 16 16:26:33 IST 2011 Thread-4: Exception org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) Wed Feb 16 16:26:33 IST 2011 Thread-4: Abort Wed Feb 16 16:26:33 IST 2011 Thread-4: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:33 IST 2011 Thread-4: new HttpGet Wed Feb 16 16:26:33 IST 2011 Thread-2: thread failure #3, System did a total of 17832 GET requests org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connectionWed Feb 16 16:26:33 IST 2011 Thread-2: Exception at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) Wed Feb 16 16:26:33 IST 2011 Thread-2: Abort Wed Feb 16 16:26:33 IST 2011 Thread-2: Killing the JVM because reaching maximum (debug purposes) failures Wed Feb 16 16:26:33 IST 2011 Thread-2: new HttpGet Wed Feb 16 16:26:34 IST 2011 Thread-1: thread failure #4, System did a total of 17940 GET requests Wed Feb 16 16:26:34 IST 2011 Thread-1: Exception Wed Feb 16 16:26:34 IST 2011 Thread-1: Abort org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:417) at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:391) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at HTTPClientTest$GetThread.run(HTTPClientTest.java:82) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
