On Sat, Oct 25, 2003 at 11:44:21AM +0100, Dave Hooper wrote: > [EMAIL PROTECTED],flaghATTIcmo6j3ic7qohg/[EMAIL PROTECTED] > sH9HWFrYxIGEw0PAgM/DFI// > > Woo, just got it. How does the spider work and how does it pick up new > sites?
It's pretty simple, really, both in theory and in practice. Basically, you "seed" the spider with a few start keys (I use all of the major index sites, including my own), and it recurses through every link it finds. You can run it either in a simple GUI, or from the command line (handy for doing periodic auto-regeneration of the html pages). The resulting pages are a little on the large site, but they certainly do look nice. :-) In the days to come, I'm going to try to clean things up some more, eliminate some of the bloat, etc. Incidentally, you may notice a decrease in the sites found over the next few days. To try and get the most accurate picture of what's really available within the unstable network, I cleaned out my datastore and restarted the spider from scratch this morning. I didn't want it showing stuff that may have been found only in my own local datastore. :-) > [EMAIL PROTECTED],yuCNX5UUQUgbRNv3Ire46w/BigAnimalHead// > Should be available on the unstable network now Yep, I'm getting it here. What a wonderfully weird and wacky site! :-) -- Conrad Sabatier <[EMAIL PROTECTED]> - "In Unix veritas" _______________________________________________ Devl mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/devl
