I am just curious how long it typically takes to crawl a site? I am crawling server side which I realize is big. I also changed the urlfilter file to accept urls with ? so I could grab all the stuff from the forums and am doing a depth of 10 but I started crawling around 9 last night and now 13 hours later, it is still going. The crawl directory is up to about 627 meg. I can't imagine how long it would take if I tried to crawl the web.
Thanks! ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id492&op=click _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
