Subjects of the emperor have been known to write crawlers specifically intended to ignore robots.txt.
The 'Robot Exclusion Standard' is voluntary; that is, not a normative requirement. Brian From: Douglas Nix [mailto:[email protected]] Sent: Friday, April 11, 2014 10:28 AM To: [email protected] Subject: Re: [PSES] PED and TPED Directives (Google Knows Everything) Anthony: 1) If you have top secret stuff on your web server in the first place, that's actually the problem and not what Google's spiders can find, and 2) That's what the robots.txt file is for. All you need to do is exclude them from those pages you don't want crawled for any reason. 3) Reference point 1. Doug On 11-Apr-2014, at 18:55, Anthony Thomson <[email protected]> wrote: John's right, "Google knows where it is", and I can't help wondering what it finds that it shouldn't. I imagine Google's crawled searches are passed through very sophisticated filtering algorithms that divert certain discovered content to a top secret office buried deeply and covertly somewhere within the Googleverse. Such content could be ultra secrect, revolutionary or illegal and what happens to it thereafter, who knows? Tony... (the Conspiricy Theorist (not really)) ----- Original Message ----- From: John Woodgate Sent: 04/10/14 04:32 PM To: [email protected] Subject: Re: [PSES] PED and TPED Directives >I seem to have difficulty locating a copy of Directive 2008/68/EC >Inland Transport of Dangerous Goods. Google knows where it is. http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2008:260:0013: 0059:en:PDF -- OOO - Own Opinions Only. With best wishes. See www.jmwa.demon.co.uk Nondum ex silvis sumus John Woodgate, J M Woodgate and Associates, Rayleigh, Essex UK - ---------------------------------------------------------------- This message is from the IEEE Product Safety Engineering Society emc-pstc discussion list. To post a message to the list, send your e-mail to <[email protected]> All emc-pstc postings are archived and searchable on the web at: http://www.ieee-pses.org/emc-pstc.html Attachments are not permitted but the IEEE PSES Online Communities site at http://product-compliance.oc.ieee.org/ can be used for graphics (in well-used formats), large files, etc. Website: http://www.ieee-pses.org/ Instructions: http://www.ieee-pses.org/list.html (including how to unsubscribe) List rules: http://www.ieee-pses.org/listrules.html For help, send mail to the list administrators: Scott Douglas <[email protected]> Mike Cantwell <[email protected]> For policy questions, send mail to: Jim Bacher: <[email protected]> David Heald: <[email protected]>
