Robots can't be "barred" or banned from a site - whether a bot observes any robots.txt or META directives is entirely at the discretion of the bot operators or developers.
At 10:50 AM 3/8/00 -0000, Brian Kelly wrote: >What do robots do if they index a web site and the resources they index >are subsequently made unavailable to robots through use of the >robots.txt file or the META ROBOTS tag? Will they remain in the index, >even though the content may have changed (which spammers could exploit) >or would they be deleted once the robot exclusion was detected? > >Thanks > >Brian Kelly > >-------------------------------------------------------------------- >Brian Kelly, UK Web Focus >UKOLN, University of Bath, BATH, England, BA2 7AY >Email: [EMAIL PROTECTED] URL: http://www.ukoln.ac.uk/ >Homepage: http://www.ukoln.ac.uk/ukoln/staff/b.kelly.html >Phone: 01225 323943 FAX: 01225 826838 >
