On 26 December 2012 13:10, Oleg Kalnichevski <[email protected]> wrote:
> > Just out of curiosity, why are using an asynchronous HTTP client for a > web crawler? I personally would consider a blocking HTTP client a much > better choice for a heavy duty web crawler. > Well... how would you manage 10-20K simultaneously opened connections with a synchronous client? We tried with that number of threads each with a DefaultHttpClient, but the problem is that as soon as there is any contention (even on logging) it slows down terribly everything.
