Mattias, I am aware of Google's policy about serving different content to search users, which is why I am have to implement their "First Click Free" program.
I will use the User-Agent but need to go a step further and verify the crawler is who they say they are by DNS. Cheers, Richard On 3/7/11 9:05 AM, "Mattias Geniar" <[email protected]> wrote: > Hi, > > I would look at the user agent to verify if it's a GoogleBot or not, as > that's more easily checked via VCL. All GoogleBots also adhere to the > correct User-Agent. > There really aren't that many users that spoof their User-Agent to gain > extra access. > > Also keep in mind that serving GoogleBot different content than actual > users will get you penalties in SEO, eventually dropping your Google > ranking. Just, FYI. > > Regards, > Mattias > > From: [email protected] > [mailto:[email protected]] On Behalf Of Lane, > Richard > Sent: maandag 7 maart 2011 15:58 > To: [email protected] > Subject: Let GoogleBot Crawl full content, reverse DNS lookup > > > I am looking into supporting Google's "First Click Free for Web Search". > I need to allow the GoogleBots to index the full content of my sites but > still maintain the Registration wall for everyone else. Google suggests > that you detect there GoogleBots by reverse DNS lookup of the requesters > IP. > > Google Desc: > http://www.google.com/support/webmasters/bin/answer.py?answer=80553 > > Has anyone done DNS lookups via VCL to verify access to content or to > cache content? > > System Desc: > Varnish 2.1.4 > RHEL 5-4 > Apache 2.2x > > - Richard _______________________________________________ varnish-misc mailing list [email protected] http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
