Re: [htdig] wildcards in robot.txt?

[Gabriele Bartolini]

Ciao!

At 12.15 06/02/2003 -0500, [EMAIL PROTECTED] wrote:

User-agent: htdig-udem
Disallow: *.pdf
Disallow: *.PDF

Is this possible with ht://Dig 3.1.6?

Well ... robots.txt is an 'international' and de-facto standard and it is not an ht://Dig's feature; I snipped these word from the 'www.robotstxt.org' site:

Note also that regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Specifically, you cannot have lines like "Disallow: /tmp/*" or "Disallow: *.gif".

I don't know whether it is a good idea to override this. What about you guys?

Ciao
-Gabriele
--
Gabriele Bartolini - Web Programmer - ht://Dig & IWA Member - ht://Check maintainer
Current Location: Prato, Tuscany, Italia
[EMAIL PROTECTED] | http://www.prato.linux.it/~gbartolini | ICQ#129221447
> find bin/laden -name osama -exec rm {} \;



-------------------------------------------------------
This SF.NET email is sponsored by:
SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
http://www.vasoftware.com
_______________________________________________
htdig-general mailing list <[EMAIL PROTECTED]>
To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe
FAQ: http://htdig.sourceforge.net/FAQ.html