Hi!

Wget 1.5.3 uses /robots.txt to skip some parts of web-site. But it
doesn't use <META NAME="ROBOTS" CONTENT="NOFOLLOW"> tag, which serves
to the same purpose.

I believe that Wget must also parse and use <META NAME='ROBOTS' ...>
tags

WBR
 Stas                          mailto:[EMAIL PROTECTED]

Reply via email to