But you give away a lot of money if you block the AdSense crawler,
because AdSense cannot deliver relevant ads, therefore the click rate
is low and also the profit.
On 17 Feb., 02:02, Josh (Google Employee) wrote:
> On Feb 16, 1:51 pm, GA wrote:
>
> > if you permit u=http in your robots.txt file, the AdSense crawler is
> > allowed to crawl and the error message should
> > disappear.http://translate.googleusercontent.com/robots.txt
>
> We correctly block robots.txt from crawling our translations of
> webpages. We don't want web crawlers to try to crawl complete
> translations of the web through our service ;-)
>
> Cheers,
> Josh
