Hello, I am crawling multiple sites, in range of hundreds, with 256 concurrent threads, and 4 conns per site at a time.
It seems that if a site is having a bad day, all the threads slow down, and this site basically clogs all the threads. basically if one site is slow, it slowly accumulates all the threads to be slow. Is there a way around this? Or if a site is giving excessive 404's - or 500's how can we avoid crawling at that run? Best Regards, C.B.
