Over a gig now, 18 hours running and still going...might just have to kill it unless this is typical.
On 7/27/05, blackwater dev <[EMAIL PROTECTED]> wrote: > I am just curious how long it typically takes to crawl a site? I am > crawling server side which I realize is big. I also changed the > urlfilter file to accept urls with ? so I could grab all the stuff > from the forums and am doing a depth of 10 but I started crawling > around 9 last night and now 13 hours later, it is still going. The > crawl directory is up to about 627 meg. I can't imagine how long it > would take if I tried to crawl the web. > > Thanks! >
