I don't think it is a-typical, because I had similar effects with crawl depth = 10.
Rgrds, Thomas --- blackwater dev <[EMAIL PROTECTED]> wrote: > Over a gig now, 18 hours running and still > going...might just have to > kill it unless this is typical. > > On 7/27/05, blackwater dev <[EMAIL PROTECTED]> > wrote: > > I am just curious how long it typically takes to > crawl a site? I am > > crawling server side which I realize is big. I > also changed the > > urlfilter file to accept urls with ? so I could > grab all the stuff > > from the forums and am doing a depth of 10 but I > started crawling > > around 9 last night and now 13 hours later, it is > still going. The > > crawl directory is up to about 627 meg. I can't > imagine how long it > > would take if I tried to crawl the web. > > > > Thanks! > > >
