Yes, I see how one is meant to look at it. But I can imagine desktop
aggregators
becoming more independent when searching for information... Perhaps
at that point
they should start reading robots.txt...
Henry
On 25 Aug 2005, at 23:12, Walter Underwood wrote:
I would call desktop clients "clients" not "robots". The
distinction is
how they add feeds to the polling list. Clients add them because of
human decisions. Robots discover them mechanically and add them.
So, clients should act like browsers, and ignore robots.txt.
Robots.txt is not very widely deployed (around 5% of sites), but it
does work OK for general web content.
wunder
--On August 25, 2005 10:25:08 PM +0200 Henry Story
<[EMAIL PROTECTED]> wrote:
Mhh. I have not looked into this. But is not every desktop
aggregator a robot?
Henry
On 25 Aug 2005, at 22:18, James M Snell wrote:
At the very least, aggregators should respect robots.txt. Doing so
would allow publishers to restrict who is allowed to pull their
feed.
- James
--
Walter Underwood
Principal Software Architect, Verity
