-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Alan
Perkins
Sent: 09 February 2001 00:58
To: [EMAIL PROTECTED]
Subject: Re: robot developers?
Here's something for discussion: The robots.txt and robots meta tag
protocols have serious flaws but there appears
Tom
[quote]
Finally, a question that exposes the worst flaw of the robots.txt
protocol:
a webmaster wishes to make all pages of a Web site, EXCEPT the home page
(i.e. /), accessible to robots; how can she do this using the robots.txt
protocol? The answer - She can't.
[unquote]
This is
On the robots side, I'd be interested to know what techniques
people are using to store URLs in the queue for later processing.
(i.e., since folks want a delay between requests, it makes sense
to have multiple input queues or use a database of some sort
to store the URLs until they are
are people out-there developing (desing, programming) robots?
I'm having one designed for my site according to my wishes.
I think it has to do with the fact that everytime someone engineers a
stroke
of genius, they keep it to themselves. Everyone wants free hand outs, but
few are willing to share their own findings.
is anybody interested in being able to grab urls out of flash movies?
i have some perl code i
So, where is the robots brainstorming taking place?
If you find the answer to your question, please let us all know here. I'm
also disappointed at the lack of content on this list.
I think it has to do with the fact that everytime someone engineers a stroke
of genius, they keep it to
