On Wed, 2012-11-14 at 19:38 +0800, Zhan Zhifeng wrote: > Hi, > > I run a crawler with httpclient. PoolingClientConnectionManager used in the > process of crawling. > I monitor the PoolStats with one monitor thread. And when my crawling > website list is zero, > and the ps_.getLeased() is always not be zero. > But i am not sure that the crawling job is finished or not. PS: > ps_.getAvailable() is zero. > > refer the > https://issues.apache.org/jira/browse/HTTPCLIENT-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > New connection pool and pooling connection manager implementations now > expose pool stats such number of leased & available connections. > Stats can be obtained either for a particular route or the entire pool. > > I also do the following method to try to release the connection in pool, > but it seems not work: > (1) _httpclient.getParams(). > setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 100000); > (2) In monitor thread, connMgr.closeExpiredConnections(); > connMgr.closeIdleConnections(30, TimeUnit.SECONDS); > > I am wandering if there is any active crawling thread using these leased > connections, > and how can i know it or how can i operate these leased connections? > And then i can finish my crawling job more accurately. > > Thanks a lot.
I think your code may be leaking connections. Just make sure that you _always_ response content (even if you do not need it) and it will automatically take care of connection release. Oleg --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
