In <[EMAIL PROTECTED]>, Jonathan Knoll <[EMAIL PROTECTED]> writes: > Do robots.txt files have stemming/wildcard abilities? > > ********************** > User-agent: * > Disallow: /cgi-bin > Disallow: /site > **********************
Yes, robots are instructed to not visit any URLs that start with the given string, so in your example /cgi-bin/test.cgi /siteindex.html would be excluded. Klaus Johannes Rusch -- [EMAIL PROTECTED] http://www.atmedia.net/KlausRusch/
