On Friday, August 26, 2005, at 04:39 AM, Eric Scheid wrote:
On 26/8/05 3:55 PM, "Bob Wyman" <[EMAIL PROTECTED]> wrote:
Remember, PubSub never does
anything that a desktop client doesn't do.
Periodic re-fetching is a robotic behaviour, common to both desktop
aggregators and server based aggregators. Robots.txt was established to
minimise harm caused by automatic behaviour, whether by excluding
non-idempotent URL, avoiding tarpits of endless dynamic links, and such
forth. While true that each of these scenarios involve crawling new
links,
the base principle at stake is to prevent harm caused by automatic or
robotic behaviour. That can include extremely frequent periodic
re-fetching,
a scenario which didn't really exist when robots.txt was first put
together.
I'm with Bob on this. If a person publishes a feed without limiting
access to it, they either don't know what they're doing, or they're
EXPECTING it to be polled on a regular basis. As long as PubSub
doesn't poll too fast, the publisher is getting exactly what they
should be expecting. Any feed client, whether a desktop aggregator or
aggregation service, that polls too fast ("extremely frequent
re-fetching" above) is breaking the rules of feed consuming
etiquette--we don't need robots.txt to tell feed consumers to slow down.
