Stemming and Wildcards in robots.txt files

I'm sorry, I am a bit new to this, so this may be a bit simplistic. Nonetheless, I must ask. Do robots.txt files have stemming/wildcard abilities? ** User-agent: * Disallow: /cgi-bin Disallow: /site ** Will this only disallow the cgi-bin and the directory

Disallows in robots.txt

So: I was looking at a robots.txt file and it had a series of disallow instructions for various user agents, and then at the bottom was a full disallow: - User-agent: aa Disallow: /cgi-bin Disallow: /stuff Disallow: /x.html User-agent: bb

FW: Stemming and Wildcards in robots.txt files

OK, then is there a way to create an internal wildcard? User-agent: * Disallow: /*/97 Disallow: /*/98 Something like this? >> Jonathan Knoll: >> >> User-agent: * >> >> Disallow: /cgi-bin >> >> Disallow: /site >> >> Klaus Johannes Rusch: >