By the way i would think to make usage of robots.txt configurable would be interesting for some users of nutch. ;-).
Is the reason known why it hangs?
Stefan
Am 26.05.2004 um 23:33 schrieb Doug Cutting:
Stefan Groschupf wrote:How ever after a around 5 hours of fetching the fetching process hangs.
I only get the message fetch-list is empty but the fetcher is not finished.
Someone notice similar things?
The RequestScheduler fetcher implementation is known to hang. Have you tried instead using Fetcher.java implementation? This didn't used to respect robots.txt, but now it does. Try using it instead, with a command like:
bin/nutch net.nutch.fetcher.Fetcher ...
Does that work better for you?
Doug
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
--------------------------------------------------------------- open technology: http://www.media-style.com open source: http://www.weta-group.net open discussion: http://www.text-mining.org
-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
