What do robots do if they index a web site and the resources they index are subsequently made unavailable to robots through use of the robots.txt file or the META ROBOTS tag? Will they remain in the index, even though the content may have changed (which spammers could exploit) or would they be deleted once the robot exclusion was detected?
Thanks Brian Kelly -------------------------------------------------------------------- Brian Kelly, UK Web Focus UKOLN, University of Bath, BATH, England, BA2 7AY Email: [EMAIL PROTECTED] URL: http://www.ukoln.ac.uk/ Homepage: http://www.ukoln.ac.uk/ukoln/staff/b.kelly.html Phone: 01225 323943 FAX: 01225 826838
