Hello, Randal,
Ironically, OS X mail flagged your post as spam, and I've only just
found it.
On Apr 14, 2006, at 5:31 PM, Randal L. Schwartz wrote:
The original Inktomi spider was written in Perl, and even available
as public
open source for a while. "Back in the day."
I've been unable to hunt down that source code. Is it archived on
any public site?
Is this for a private network? Please tell me you're not trying to
build Yet
Another Spider to visit my site!
This specific spider wouldn't visit stonehenge.com, as the subject
matter wouldn't be relevant. But I take it that you mean "my site"
in a collective sense, and that you're objecting philosophically to
the potential increase in average server load if the web spider
population explodes. Am I reading you right?
I'm afraid I think that web spiders are going to multiply regardless
of what I do personally. At this time, I'm dedicated to KinoSearch
and I have zero interest in publishing and maintaining a "Putch", but
Nutch exists, it's getting better, and other competitors to it are
going to appear.
Some of these spiders are going to obey robots.txt, some won't.
Certainly any of the ones I write will.
If so, please use the Google or Yahoo API to
leverage the fact that they've already done an excellent job of
visiting a few
thousand URLs, including 40,000 images. You don't need to come here.
Another option is the Alexa database, from which you can grab data
for a per-GB fee.
http://websearch.alexa.com/docs/price_guide.html
However, the Alexa crawl data may not contain the pages you want, or
update frequently enough to meet your needs.
If you want your web search site to compete with Google/Yahoo, it's
doubtful that you would want to farm out the task of spidering. Of
course that's true for major Google competitors such as
search.msn.com and ask.com. More to the point, though, is that you
may not want to farm out spidering even if you are competing only in
a narrow niche.
http://wiki.apache.org/nutch/PublicServers
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
