On Thu, 2012-12-27 at 12:30 -0800, Sebastiano Vigna wrote: > On 26 December 2012 13:10, Oleg Kalnichevski <[email protected]> wrote: > > > > > Just out of curiosity, why are using an asynchronous HTTP client for a > > web crawler? I personally would consider a blocking HTTP client a much > > better choice for a heavy duty web crawler. > > > > Well... how would you manage 10-20K simultaneously opened connections with > a synchronous client?
But why would you want a web crawler to have 10-20K simultaneously opened connections in the first place? Oleg > We tried with that number of threads each with a > DefaultHttpClient, but the problem is that as soon as there is any > contention (even on logging) it slows down terribly everything. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
