Paul Wratt <paul.wr...@gmail.com> writes: > if it does not obey - server admins will ban it > > the work around: > 1) get single html file first - edit out meta tag - re-get with > --no-clobber (usually only in landing pages) > 2) empty robots.txt (or allow all - search net) > > possible solutions: > A) command line option > B) ./configure --disable-robots-check
you can specify -e robots=off to wget at runtime. Giuseppe
