> Remember, PubSub never does > anything that a desktop client doesn't do.
Bob: What about FeedMesh? If I ping blo.gs, they pass that ping along to you, and PubSub fetches my feed, then PubSub is doing something a desktop client doesn't do. It's following a link found in one place and retrieving/indexing/polling a document somewhere else... sounds distinctly spidery to me. But honestly, I'm not interested in nit-picking anyone's definition of "robot". To me, it's a matter of being friendly to the people providing all that content... if they make a point of telling me to stay away, I'm going to stay away. Let's say I'm absolutely convinced that my republishing aggregator isn't a spider, for whatever reason. Fine, I'm going to ignore any "*" directives in robots.txt. But I'm not going to ignore the file entirely, because if someone goes to the trouble to add an entry specifically for me, then that's a hint-and-a-half that I need to leave him alone. We've got a mechanism that allows any user with his own domain and a text editor to tell us whether or not he wants us messing with his stuff. I think it's foolish to ignore that. -- Roger Benningfield
