Nah, they would have e-mailed me directly by now. I used to work with them at Inktomi.
How about dropping them an e-mail to invite them here?
Yahoo limits crawler access to its own site. I haven't tried in the last 9 or 10 months, but the way it was back then, if you crawled the message boards, the crawler's IP address would be blocked for increasingly long time periods -- a day, two days, etc. I tried slowing down our gathering, but couldn't find a speed at which they wouldn't eventually block it. And of course they never responded to any questions about what they'd consider acceptable.
And yet, their own servers don't seem to have a robots.txt that defines any limitations. Sure would be nice if *they* would tell *us* what's acceptable when crawling Yahoo!
Nick
-- Nick Arnett Director, Business Intelligence Services LiveWorld Inc. Phone/fax: (408) 551-0427 [EMAIL PROTECTED]
_______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
