Quit top posting.

On Mon, Nov 29, 2010 at 9:55 PM, Ron Piggott <ron.pigg...@actsministries.org
> wrote:

> My issue with the user agent is unresolved.  I need to do more research to
> see how AWSTATS distinguishes between a robot crawling the site and a web
> page user and set the user-agent accordingly.


AWSTATS probably users a knowledge base for known bots, I'm not sure. If
that's the case, you can just set your User-Agent to a known and see how
that goes.

Look for Googlebot, Majestic, Ask.com (now dead - probably a good pick),
MSNBot here: http://www.user-agents.org/

As for setting the User-Agent in your request, I like to use this cUrl
snippet (based on a note at curl's manual page):


  $sUrl = 'www.example.com/';
  $sUserAgent = 'Googlebot/2.1 (+http://www.googlebot.com/bot.html)';

  $hCurl = curl_init();
  curl_setopt ($hCurl, CURLOPT_RETURNTRANSFER, TRUE);
  curl_setopt ($hCurl, CURLOPT_URL, $sUrl);
  curl_setopt ($hCurl, CURLOPT_CONNECTTIMEOUT, 120);
  curl_setopt ($hCurl, CURLOPT_TIMEOUT, 120);
  curl_setopt ($hCurl, CURLOPT_USERAGENT, $sUserAgent);

  $sContent = curl_exec($hCurl);

Thiago Henrique Pojda
+55 41 8856-7925

Reply via email to