just make sure you have "Mozilla/5.0" at the front :)
----- Original Message -----
From: "Insurance Squared Inc." <[EMAIL PROTECTED]>
To: <[email protected]>
Sent: Friday, December 09, 2005 2:55 PM
Subject: User Agent
What should I be using for a user agent in the crawler? We just tried
crawling a government site and if we leave the user agent set to nutch,
we get the crawl. When I change it, I'm getting blocked with an error
about the user agent not being supported. It seems that I should be
changing the user agent to something that identifies my site - shouldn't
I? But that might limit my crawling ability.
What are others setting their user agent to?
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
