hi, Oleg
How can I "_always_ response content "?
I have do these below:
HttpEntity entity = response.getEntity();
if (entity != null && header_[0].getValue().
contains("text/html")) {
String html_ = EntityUtils.toString(entity, "UTF-8");
EntityUtils.consume(entity);
} else {
EntityUtils.consume(entity);
}
2012/11/15 Oleg Kalnichevski <[email protected]>
> On Wed, 2012-11-14 at 19:38 +0800, Zhan Zhifeng wrote:
> > Hi,
> >
> > I run a crawler with httpclient. PoolingClientConnectionManager used in
> the
> > process of crawling.
> > I monitor the PoolStats with one monitor thread. And when my crawling
> > website list is zero,
> > and the ps_.getLeased() is always not be zero.
> > But i am not sure that the crawling job is finished or not. PS:
> > ps_.getAvailable() is zero.
> >
> > refer the
> >
> https://issues.apache.org/jira/browse/HTTPCLIENT-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> > New connection pool and pooling connection manager implementations now
> > expose pool stats such number of leased & available connections.
> > Stats can be obtained either for a particular route or the entire pool.
> >
> > I also do the following method to try to release the connection in pool,
> > but it seems not work:
> > (1) _httpclient.getParams().
> > setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 100000);
> > (2) In monitor thread, connMgr.closeExpiredConnections();
> > connMgr.closeIdleConnections(30, TimeUnit.SECONDS);
> >
> > I am wandering if there is any active crawling thread using these leased
> > connections,
> > and how can i know it or how can i operate these leased connections?
> > And then i can finish my crawling job more accurately.
> >
> > Thanks a lot.
>
> I think your code may be leaking connections. Just make sure that you
> _always_ response content (even if you do not need it) and it will
> automatically take care of connection release.
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
--
网络与媒体实验室·浙江大学
占志峰
13588163850
