Hi, I run a crawler with httpclient. PoolingClientConnectionManager used in the process of crawling. I monitor the PoolStats with one monitor thread. And when my crawling website list is zero, and the ps_.getLeased() is always not be zero. But i am not sure that the crawling job is finished or not. PS: ps_.getAvailable() is zero.
refer the https://issues.apache.org/jira/browse/HTTPCLIENT-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel New connection pool and pooling connection manager implementations now expose pool stats such number of leased & available connections. Stats can be obtained either for a particular route or the entire pool. I also do the following method to try to release the connection in pool, but it seems not work: (1) _httpclient.getParams(). setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 100000); (2) In monitor thread, connMgr.closeExpiredConnections(); connMgr.closeIdleConnections(30, TimeUnit.SECONDS); I am wandering if there is any active crawling thread using these leased connections, and how can i know it or how can i operate these leased connections? And then i can finish my crawling job more accurately. Thanks a lot.
