On Thursday 08 August 2013 05:51:08 atar wrote:
wget download the robots.txt file although the '-R txt' option was
specified.

wget -e robots=no <url>

or to switch if off forever put
        robots=no
into /etc/wgetrc or ~/.wgetrc.

Regards, Tim

Thanks!!!

Reply via email to