> From: Jens R�sner [mailto:[EMAIL PROTECTED]] > Subject: Re: robots.txt
> > or your indexing mech might loop on it, or crash the > server. who knows. > I have yet to find a site which forces wGet into a "loop" as you said. > Others on the list probably can estimate the theoretical likelyhood of > such events. We do have some sites like that (dynamic pages with a possible infinite depth), although not accessible to the whole internet. So at least the probability is definitevely >0 :) Heiko Herold -- -- PREVINET S.p.A. [EMAIL PROTECTED] -- Via Ferretto, 1 ph x39-041-5907073 -- I-31021 Mogliano V.to (TV) fax x39-041-5907472 -- ITALY
