Interestingly, not all the search engines pay attention to robots.txt. Library of Congress uses Archive.org, and they were throwing a lot of errors on my site by pulling up pages without url parameters, even though those pages were in my robots.txt. I finally got a hold of someone there and they basically told me they ignore robots.txt because so many people tell them not to archive images.
On 7/4/07 7:17 AM, "Will Tomlinson" <[EMAIL PROTECTED]> wrote: > Can you use a robots.txt file to tell google to ignore these pages? > > Will > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Download the latest ColdFusion 8 utilities including Report Builder, plug-ins for Eclipse and Dreamweaver updates. http;//www.adobe.com/cfusion/entitlement/index.cfm?e=labs%5adobecf8%5Fbeta Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:286035 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4
