You are probably right. The site was released in a redesigned format at the beginning of February. At that time, they no longer used wireless.metacritic.com, but instead www.metacritic.com/avantgo. This new URL worked until this week.It's down (wireless.metacritic.com), and probably because they noticed a lot of non-AvantGo UserAgent strings hitting the site. This is what happens when people pound the site with wget, Plucker, perl, etc. and forget to change the UserAgent string passed to the server.
The explanation provided to me by someone at C-Net was not related to 'site pounding' at all. They knew that the spiders were stripping advertising as well. Many sites have to justify feeds to their advertisers. Here is a quote from a person at one major site:
We prefer that people use AvantGo to view our mobile-friendly content, as we can better gauge (anonymous) usage of that version of the site. Advertising is our major revenue stream, so knowing at least the size of our audience is very important. We're not in favor of one operating system or device. We are in favor of the most people viewing our content as possible, provided that we continue to make money for that viewing.
JPluck allows you to change the user agent, but doing does not help for Metacritic. Many sites, such as C-Net News, have moved to avantgo.com.com and you have to know the 'secret' URL to access them now.But how do you faik so that your IP looks like it comes from Avant Go? Changing the user agent string would not fool them, would it?
Ed
