Thanks Roland! -----Original Message----- From: Roland Whitehead [mailto:rol...@quru.com] Sent: Wednesday, 17 February 2016 6:20 PM To: user@jspwiki.apache.org Subject: Re: robots.txt files and JSPWiki
On 17/02/2016 at 01:39, Jason Morris wrote: > What would be the proper config of robots.txt to allow only Google to crawl a > JSPWiki installation and nothing else? It's all in Google's robots.txt documentation https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt However, something like this might work for you: # robots.txt User-agent: Googlebot Disallow: User-agent: googlebot-image Disallow: User-agent: googlebot-mobile Disallow: User-agent: * Disallow: / Sitemap: http://www.yoursite.com/sitemap.gz Roland -- QURU Ltd, London, UK http://quru.com
