[Robots] Avoiding Bot-Bait pages and wspoison pages

Hello everyone , How do you guys create your web crawler in such a way that it would step over bot bait pages like WSPosion? Do you simply include them in a list of urls to avoid ? or do you keep track of web sites with unusually large amounts of web page such as a web site with about 200

[Robots] how does the nytimes feel about it when your robot checks their news articles

Hi how does news web sites like the nytimes, washington times, cnet news etc feel about robots that come and take a look at their news items ? sincerely yours, stephen __ Do You Yahoo!? Yahoo! Movies - coverage of the 74th Academy Awards®