https://www.schneier.com/blog/archives/2013/06/more_on_feudal.html
FWIW before Google when there were lots of search engines, one company bragged that you could toggle to ignore robots.txt. It was considered highly unethical and I think the company suffered. robots.txt is just like the yellow line on the highway. Having everyone honor it is what keeps you from having a 50 mph head on collision but the line won't do anything to prevent it.
_______________________________________________ gnhlug-discuss mailing list gnhlug-discuss@mail.gnhlug.org http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
