NETWORK WORLD NEWSLETTER: MARK GIBBS ON WEB APPLICATIONS 11/22/04 Today's focus: Creating and enforcing robot exclusion
Dear [EMAIL PROTECTED], In this issue: * Where the Robot Exclusion Standard falls short and Robotcop ��fills the gap * Links related to Web Applications * Featured reader resource _______________________________________________________________ This newsletter is sponsored by Intel IT Productivity; Increasing ROI Learn how to effectively measure employee productivity, manage IT investments and reduce the Total Cost of Ownership in enterprise data management. Visit Intel's IT Productivity center. Click here to download white papers, books and IDC Research. http://www.fattail.com/redir/redirect.asp?CID=88615 _______________________________________________________________ NW'S RESEARCH CENTER ON SPAM Go to NW's Research Center on spam and find our in-depth review of 16 anti-spam products, our spam calculator to determine how much spam is costing your enterprise each year, the latest spam news, advice on how to fight spam and more. For the latest on spam click here: http://www.fattail.com/redir/redirect.asp?CID=88680 _______________________________________________________________ Today's focus: Creating and enforcing robot exclusion By Mark Gibbs If there's one thing that every commercial Web site wants, it is for the search engine spiders to crawl their sites and make them findable. But sites don't always want to have their entire contents indexed or possibly be indexed by every spider. This is the reason we have Robot Exclusion Standard (see editorial links below). The RES is implemented as a file named robots.txt in the server's root that specifies which spiders can go to which subdirectories. Even though the protocol is pretty simple, there are a number of online services that will generate correctly formed robots.txt files or check existing files for correctness. Surprisingly, despite the simplicity of the standard some sites apparently still get it wrong. According to the Our Big Crawl site, which examined the robots.txt files from more than 75,000 sites, some 5% used bad style and around 2% were so badly formed that "they would not be recognized by any spider." The problem with the standard is that it can't enforce compliance. This means that spiders used by spammers to harvest e-mail addresses will just ignore any robots.txt file, as will spiders used by any organization that doesn't care about your wishes. The solution is to use Robotcop (tag: "robots.txt: It's the Law"). Robotcop operates by checking all requests to a site - if the requests don't pass the tests then Robotcop doesn't allow the request to proceed and can ban the requesting IP address from all access (the IP address is removed from the list when it sends no further requests for a configurable period). Robotcop is written in C for the Apache 1.3 server for Red Hat and FreeBSD with plans to port the code to IIS and Apache 2.x in the future. RELATED EDITORIAL LINKS The Web Robots Pages http://www.robotstxt.org/ Robot Control Code Generation Tool http://www.mcanerin.com/search-engine/robots-txt.htm#generator Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi The Robots.txt Our Big Crawl http://www.searchengineworld.com/misc/robots_txt_crawl.htm Robotcop http://www.robotcop.org/ _______________________________________________________________ To contact: Mark Gibbs Mark Gibbs is a consultant, author, journalist, and columnist and he writes the weekly Backspin and Gearhead columns in Network World. We'll spare you the rest of the bio but if you want to know more, go to <http://www.gibbs.com/mgbio>. Contact him at <mailto:[EMAIL PROTECTED]> _______________________________________________________________ This newsletter is sponsored by Intel IT Productivity; Increasing ROI Learn how to effectively measure employee productivity, manage IT investments and reduce the Total Cost of Ownership in enterprise data management. Visit Intel's IT Productivity center. Click here to download white papers, books and IDC Research. http://www.fattail.com/redir/redirect.asp?CID=88614 _______________________________________________________________ ARCHIVE LINKS Archive of the Web Applications newsletter: http://www.nwfusion.com/newsletters/web/index.html _______________________________________________________________ FEATURED READER RESOURCE NW FUSION PARTNERS' SITES NOW AVAILABLE Network World Fusion Partners is a collaborative effort between Network World and sponsoring Partner companies. Each microsite contains best-of-breed information as well as custom content not found anywhere else, including a custom email newsletter and special offers. It is current, top-of-mind information that is readily accessible and bundled into one comprehensive package. Visit the NWFusion Partner sites to learn about storage solutions, network access solutions, optical networking and more. Visit NWFusion Partners at: <http://www.nwfusion.com/go/nwprr> _______________________________________________________________ May We Send You a Free Print Subscription? You've got the technology snapshot of your choice delivered at your fingertips each day. Now, extend your knowledge by receiving 51 FREE issues to our print publication. Apply today at http://www.subscribenw.com/nl2 International subscribers click here: http://nww1.com/go/circ_promo.html _______________________________________________________________ SUBSCRIPTION SERVICES To subscribe or unsubscribe to any Network World e-mail newsletters, go to: <http://www.nwwsubscribe.com/Changes.aspx> To unsubscribe from promotional e-mail go to: <http://www.nwwsubscribe.com/Preferences.aspx> To change your e-mail address, go to: <http://www.nwwsubscribe.com/ChangeMail.aspx> Subscription questions? Contact Customer Service by replying to this message. This message was sent to: [EMAIL PROTECTED] Please use this address when modifying your subscription. _______________________________________________________________ Have editorial comments? Write Jeff Caruso, Newsletter Editor, at: <mailto:[EMAIL PROTECTED]> Inquiries to: NL Customer Service, Network World, Inc., 118 Turnpike Road, Southborough, MA 01772 For advertising information, write Kevin Normandeau, V.P. of Online Development, at: <mailto:[EMAIL PROTECTED]> Copyright Network World, Inc., 2004 ------------------------ This message was sent to: [EMAIL PROTECTED]
