re
On Thu, Jun 23, 2011 at 10:09:25AM +0200, Martin Hepp wrote:
> Yes, WebID is out of question a good thing. I am not entirely sure, though,
> that you can make it a mandatory requirement for access to your site, because
> if a few major consumers do not use WebID for their crawlers, site-owners
> cannot block anonymous crawlers.
Google, Bing and Yahoo Authenticate themself via DNS: Do a reverse lookup for
the IP, check for some well known domains and then do a forward lookup of the
hostname and check if it matches the IP. Much simpler to implement than WebID.
config = {
'Googlebot':['googlebot.com'],
'Mediapartners-Google':['googlebot.com'],
'msnbot':['live.com','msn.com','bing.com'],
'bingbot':['live.com','msn.com','bing.com'],
'Yahoo! Slurp':['yahoo.com','yahoo.net']
}
Regards,
Michael Brunnbauer
--
++ Michael Brunnbauer
++ netEstate GmbH
++ Geisenhausener Straße 11a
++ 81379 München
++ Tel +49 89 32 19 77 80
++ Fax +49 89 32 19 77 89
++ E-Mail [email protected]
++ http://www.netestate.de/
++
++ Sitz: München, HRB Nr.142452 (Handelsregister B München)
++ USt-IdNr. DE221033342
++ Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++ Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
