I was wondering what others are setting the max number of fetches per host to. I'm currently doing between 500-1000. Do you not set this at all and just set a timeout between fetches to the same host?
Howie
- RE: URL containing "?", "&" and &q... Richard Braman
- RE: URL containing "?", "&" a... Richard Braman
- Re: URL containing "?", "&" a... Vertical Search
- Re: URL containing "?", "&" a... Marko Bauhardt
- Re: URL containing "?", "&" a... Vertical Search
- Re: URL containing "?", "&" a... Vertical Search
- crawling etiquette Howie Wang
