> ... Perhaps it should be one of those things that one can do > oneself if one must but is generally frowned upon (like making a > version of wget that ignores robots.txt).
Damn. I was only joking about ignoring robots.txt, but now I'm thinking[1] there may be good reasons to do so... maybe it should be in mainline wget. T [1] http://web.archive.org/web/20041013225557/http://www.differentstrings.info/archives/002813.html
