On Thursday 15 January 2009 13:57:39 Piero Sartini wrote:
> > But, the problem with this was that wget honors
> > the new robots.txt file which means wget refuses to pull it down.
>
> wget should ignore the robots.txt with the option
> -erobots=off
>
> (or add the line robots=off in .wgetrc)
>
Oh,
On Thursday 15 January 2009 13:57:39 Piero Sartini wrote:
> > But, the problem with this was that wget honors
> > the new robots.txt file which means wget refuses to pull it down.
>
> wget should ignore the robots.txt with the option
> -erobots=off
>
Thanks Piero, using this argument, I was able t
It sounds good, thanks for setting this up Wes.
musachy
On Sat, Jan 17, 2009 at 12:51 PM, Wes Wannemacher wrote:
> On Thursday 15 January 2009 13:57:39 Piero Sartini wrote:
>> > But, the problem with this was that wget honors
>> > the new robots.txt file which means wget refuses to pull it down.
