You should never have multiple threads per host or ip unless you own it or have contacted the webmaster. Also, it is more polite to fetch only one record every few seconds.brian4 <[email protected]> schreef:I think it is because I am crawling a single host, eventually I think it throttles the crawler's connections and returns 503 for connection attempts.
I can work around this by crawling in smaller batches and pausing for a bit in between batches (and probably also by reducing threads or delaying connections per host), but since I can't guarantee I'll never encounter a 503 again I wanted to be sure it can correctly handle it by re-fetching it in a subsequent round of crawling. -- View this message in context: http://lucene.472066.n3.nabble.com/How-to-have-nutch-2-retry-503-errors-tp4123311p4123475.html Sent from the Nutch - User mailing list archive at Nabble.com.
