On Mon, Nov 20, 2023 at 1:58 PM American Citizen <[email protected]> wrote:
> > I read that using the option -e robots=off is considered rude.. is that > generally so? > > yes, that is generally the case. you are explicitly, intentionally, working against the intentions of the site administrator. it's like riding a skateboard right past a "no skateboarding" sign. the sign is there for a reason, in the same way that the robots file is there for a reason. that reason may or may not be relevant to what you're actually doing. you can't know whether it is or not, so a nice person simply assumes that it applies, and abides by the rule. a rude person flagrantly disregards the rule or even finds motivation to do the opposite. most of us fall somewhere in between those two poles most of the time :) -wes
