The white paper contains the very strange statement [quote] Finally, a question that exposes the worst flaw of the robots.txt protocol: a webmaster wishes to make all pages of a Web site, EXCEPT the home page (i.e. "/"), accessible to robots; how can she do this using the robots.txt protocol? The answer - "She can't". [unquote]
This is nonsense, what's happening here is that the Webmaster doesn't understand websites and has failed to distinguish between the default page and the home page. If she wants to allow all pages except the homepage, she can write a disallow line that explicitly excludes the home page using it's full path. She can then have the default page be a client-side redirection to the home page, so that users other than robots will get the impression that the home page is the default page. No fault with the robots.txt protocol here. Tom Thomson > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Alan > Perkins > Sent: 09 February 2001 00:58 > To: [EMAIL PROTECTED] > Subject: Re: robot developers? > > > Here's something for discussion: The robots.txt and robots meta tag > protocols have serious flaws but there appears to be no concerted > effort to > fix them. > > We've published a white paper discussing the flaws in detail at > > www.ebrandmanagement.com/whitepapers/ > > Anyone on this list is welcome to read the white paper and start a > discussion about it. You will need the following user name and password: > > User name: [EMAIL PROTECTED] > Password: [EMAIL PROTECTED] > > Enjoy... > > Alan Perkins, Chief Technology Officer > e-Brand Management Limited >
