Chris, Stew,This was always going to be controversial. It's a very tough balancing act. In our view, we're not serving additional content to seed the search engines, and so it's reasonable. We are removing content that users find useful, but which might make it harder for the search engine to make a good judgement about the site overall. Yahoo explains why this is desirable:
Webpages often include headers, footers, navigational sections, repeated boilerplate text, copyright notices, ad sections, or dynamic content that is useful to users, but not to search engines. Webmasters can apply the "robots-nocontent" attribute to indicate to search engines any content that is extraneous to the main unique content of the page.A few blogs have picked up on examples of prominent sites implementing cloaking to different extents, such as http://www.seroundtable.com/archives/021504.html and http://www.seoegghead.com/blog/seo/the-google-cloaking-hypocrisy-p32.html. Then there's also the First Click Free approach (http://www.google.com/support/webmasters/bin/answer.py?answer=74536), which many people might feel is a bit borderline.
I agree many people could use the code below to try to boost their search engine results by including lots of keywords or links, but I'm confident that there are many legitimate reasons to do this. I'd love not to do this in Varnish / HTTP, but there don't appear to be other widely supported solutions. In this case (and addressing Chris' point) it's not possible to use robots.txt as we're not trying to block the entire page, just a subset of it. There are ways of hinting to a Google Search Appliance to turn off indexing of a portion of the page (googleon/off tags, see http://perishablepress.com/press/2009/08/23/tell-google-to-not-index-certain-parts-of-your-page/), but these aren't supported by the normal googlebot, nor other search engines. Yahoo has a robots-nocontent class that can be added to HTML elements, but again, it's a single solution for just one search engine (http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-14.html). I've heard of (but can't find a link) a discussion to add a specific attribute to any HTML element to the new HTML5 standard, but that this wasn't adopted.
Someone reading this might know a magic answer, but in the meantime, we'll be making minor page alterations to help ensure users find relevant results when searching Google, even if that involves a little cloaking to suppress a small portion of the page.
Rob Chris Hecker wrote:
On that note, why not use robots.txt and a clear path name to turn off bots for the lists?Chris On 2010/08/11 08:25, Stewart Robinson wrote:Hi, Whilst this looks excellent and I may use it to serve different content to other types of users I think you should read, if you haven't already, this URL which discourages this sort of behaviour.http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=66355Great VCL though! Stew On 11 August 2010 16:20, Rob S<[email protected]> wrote:Michael Loftis wrote:--On Tuesday, August 10, 2010 9:05 PM +0100 Rob S<[email protected]>wrote:Hi, On one site we run behind varnish, we've got a "most popular" widgetdisplayed on every page (much like http://www.bbc.co.uk/news/). However, we have difficulties where this pollutes search engines, as searches fora specific popular headline tend not to link directly to the article itself, but to one of the index pages with high Google pagerank or similar. What I'd like to know is how other Varnish users might have served different ESI content based on whether it's a bot or not.My initial idea was to set an "X-Not-For-Bots: 1" header on the URL thatgenerates the most-popular fragment, then do something like (though untested):ESI goes through all the normal steps, so a<esi:includesrc="/esi/blargh"> is fired off starting with vcl_receive looking just exactly like the browser had hit the cache with that as the req.url -- the entire req object is the same -- i am *not* certain that headers you've added get propogated as I've not tested that (and all of my rules are builton the assumption that is not the case, just to be sure) So there's no need to do it in vcl_deliver, in fact, you're far betterhandling it in vcl_recv and/or vcl_hash (actually you really SHOULD handle it in vcl_hash and change the hash for these search engine specific objectselse you'll serve them to regular users)... for example -- assume vcl_recv sets X-BotDetector in the req header... (not tested):: sub vcl_hash { // always take into account the url and host set req.hash += req.url; if (req.http.host) { set req.hash += req.http.host; } else { set req.hash += server.ip; } if(req.http.X-BotDetector == "1") { set req.hash += "bot detector"; } }You still have to do the detection inside of varnish, I don't see any way around that. The reason is that only varnish knows who it's talking to, and varnish needs to decide which object to spit out. Working properly what happens is essentially the webserver sends back a 'template' for the pagecontaining the page specific stuff, and pointers to a bunch of ESIfragments. The ESI fragments are also cache objects/requests...So what happens is the cache takes this template, fills in ESI fragments (from cache if it can, fetching them if it needs to, treating them just as if the webbrowser had run to the ESI url)This is actually exactly how I handle menu's that change based on a users authentication status. The browser gets a cookie. The ESI URL is formed as either 'authenticated' 'personalized' or 'global' -- authenticated means it varies only on the clients login state, personalized takes into account the actual session we're working with. And global means everyone gets the same cache regardless (we strip cookies going into these ESI URLs and coming from these ESI URLs in the vcl_recv/vcl_fetch code, the vcl_fetch code looks for some special headers set that indicate that the recv has decided it needs to ditch set-cookies -- this is mostly a safety measure to prevent a sessionsticking to a client it shouldn't due to any bugs in code) The basic idea is borrowed from <http://varnish-cache.org/wiki/VCLExampleCachingLoggedInUsers> and <http://varnish-cache.org/wiki/VCLExampleCacheCookies> HTH!Thanks. We've proved this works with a simple setup: sub vcl_recv { .... // Establish if the visitor is a search engine: set req.http.X-IsABot = "0";if (req.http.user-agent ~ "Yahoo! Slurp") { set req.http.X-IsABot ="1"; }if (req.http.X-IsABot == "0"&& req.http.user-agent ~ "Googlebot") {set req.http.X-IsABot = "1"; }if (req.http.X-IsABot == "0"&& req.http.user-agent ~ "msnbot") { setreq.http.X-IsABot = "1"; } .... } ... sub vcl_hash { set req.hash += req.url; if (req.http.host) { set req.hash += req.http.host; } else { set req.hash += server.ip; } if (req.http.X-IsABot == "1") { set req.hash += "for-bot"; } else { set req.hash += "for-non-bot"; } hash; }The main HTML has a simple ESI, which loads a page fragment whose PHP reads:if ($_SERVER["HTTP_X_ISABOT"]) { echo "<!-- The list of popular posts is not displayed to search engines -->"; } else { // calculate most popular echo "The most popular article is XYZ"; } Thanks again.
_______________________________________________ varnish-misc mailing list [email protected] http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
