Throttling should be built right into your web server. However wouldn't you do better to throttle the bots if they are csusing problems?
Bing webmaster blog says that they support adding a crawl-delay parameter to your robots.txt file to throttle the bingbot. User-agent: msnbot Crawl-delay: 1 There's a bit more explanation in the webmaster FAQ PDF http://download.microsoft.com/download/4/5/4/454C13D4-D94D-4B54-8E46-FE403DF7632B/WMC_FAQ.pdf These other links might be helpful as well: http://www.bing.com/toolbox/webmasters http://www.bing.com/community/webmaster/f/12252/t/651373.aspx I'm sure you csn do thr same with all bots by various methods. Russ Michaels www.michaels.me.uk On 21 Jun 2013 00:39, "Richard Steele" <[email protected]> wrote: > > We're caching parts of some key pages, but unfortunately some of the data > has to be live. When the bots start hitting us, our CF8 server can't keep > up and we get long running threads. > > Is there an easy way to throttle url page requests? Perhaps throw up a > note say that the server is busy, please retry in a bit. Thanks! > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:356031 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
