In a message dated 5/4/2006 11:12:52 A.M. Central Standard Time, [EMAIL PROTECTED] writes:
Hmmm. I get page not found on that >> Hmmm, hmmm, hmmm..... If you are reading this document, then your website has likely been touched by our web crawler. We have included some information below to help you understand what a crawler is, how you can deal with crawlers, and how to contact us. Some people are surprised when a crawler visits their site regularly, downloading pages. There are many groups with crawlers which crawl the web, and a code of conduct exists to ensure that crawlers and web sites can cooperate to achieve their respective goals. Being responsible professionals, we are very anxious to make sure that webmasters are not inconvenienced by our crawling activities, and we only wish to use publicly available data. Therefore, we abide by the Robots Exclusion Standard (see _http://www.robotstxt.org/wc/exclusion.html_ (http://www.robotstxt.org/wc/exclusion.html) ), but more importantly we subscribe to the notion of being good citizens in our use of the Internet. We will therefore do our best to make sure that nobody is inconvenienced by our crawling activities. For more information please refer to _http://www.almaden.ibm.com/WebFountain_ (http://www.almaden.ibm.com/webfountain/) ------------------------------------------------------------------ What is a Crawler? ------------------------------------------------------------------ A Crawler (which may also be called a robot, spider, or bot) is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. For more information on Crawlers and the standards of crawling which we follow, you can visit the WebRobots FAQ _(http://www.robotstxt.org/wc/robots.html)_ (http://www.robotstxt.org/wc/robots.html) . ------------------------------------------------------------------ How Do I Prevent My Website or Parts of My Website From Being Crawled by Your Crawler? ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
