> Tony Godshall wrote: > >> ... Perhaps it should be one of those things that one can do > >> oneself if one must but is generally frowned upon (like making a > >> version of wget that ignores robots.txt). > > > > Damn. I was only joking about ignoring robots.txt, but now I'm > > thinking[1] there may be good reasons to do so... maybe it should be > > in mainline wget. > > Actually, it is. "-e robots=off". :) > > This also turns off obedience to the "nofollow" attribute sometimes > found in meta and a tags.
Ah, my ignorance is showing. I stand corrected.
