>> That's not a typical web crawler, and obviously not what I meant.
>>Such databases already exist (e.g. bugmenot) but using them to rip a
>>page is definitely abusive.
Not abusive at all. It's a public service.
>> Think Google, not rip-off.
Go to news.google.com and you will see many results that say things like
Kansas City Star (subscription)
So the Google crawler does indeed subscribe to some registration-required sites
and crawl them.
_______________________________________________
Boston-pm mailing list
[EMAIL PROTECTED]
http://mail.pm.org/mailman/listinfo/boston-pm
