How ever after a around 5 hours of fetching the fetching process hangs. I only get the message fetch-list is empty but the fetcher is not finished. Someone notice similar things?
The RequestScheduler fetcher implementation is known to hang. Have you tried instead using Fetcher.java implementation? This didn't used to respect robots.txt, but now it does. Try using it instead, with a command like:
bin/nutch net.nutch.fetcher.Fetcher ...
Does that work better for you?
Doug
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
