Unfortunately crawling freenet via HTTP will have the main effect of DoSing your freenet node, because every web download takes up a thread, and we therefore limit parallel HTTP downloads to 24-36. Ideally you'd want a real FCP spider; there must be one out there somewhere.
On Fri, Oct 31, 2003 at 05:58:54PM +1300, David McNab wrote: > On Fri, 2003-10-31 at 17:11, tripolar wrote: > > Hello all > > > > I need a program to crawl links in freenet to get sites in cache before > > I need them. I get frustrated with the speed of freenet and dead links. > > I have used freenet on windows & linux and just installed freenet last > > night on this winbox. I spent hours clicking on links just trying to > > pull them in. > > Anything I can do other than manually clicking on all links? > > A winbox, eh? > > Well, there are dozens of freeware and shareware (crack available) > windoze programs to recursively download websites. Sites like > www.tucows.com, www.nonags.com, www.shareware.com etc list them by the > score. > > You might need to try a few until you hit on one which doesn't molest > the '//' in freenet URIs. > > Once you choose a site downloader program, you'd need to point it at: > - http://localhost:8888/[EMAIL PROTECTED]/TFE// > - http://localhost:8888/[EMAIL PROTECTED]/YoYo// > or, point it at whatever site(s) you need, observing the above URL > syntax. Be sure to disable the timeout (or extend it to an hour or so), > because these crawler progs are used to web performance. > > One last thing - these crawler progs won't be able to tell one freesite > from another, since it will perceive all freesites as part of the same > 'site' at http://localhost:8888/. So take care to set a pattern match > requirement (unless you want the crawler to suck the whole freenet). > > Cheers > David > > _______________________________________________ > Support mailing list > [EMAIL PROTECTED] > http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so.
signature.asc
Description: Digital signature
_______________________________________________ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
