Unfortunately crawling freenet via HTTP will have the main effect of
DoSing your freenet node, because every web download takes up a thread,
and we therefore limit parallel HTTP downloads to 24-36. Ideally you'd
want a real FCP spider; there must be one out there somewhere.

On Fri, Oct 31, 2003 at 05:58:54PM +1300, David McNab wrote:
> On Fri, 2003-10-31 at 17:11, tripolar wrote:
> > Hello all
> > 
> > I need a program to crawl links in freenet to get sites in cache before 
> > I need them. I get frustrated with the speed of freenet and dead links. 
> > I have used freenet on windows & linux and just installed freenet last 
> > night on this winbox. I spent hours clicking on links just trying to 
> > pull them in.
> > Anything I can do other than manually clicking on all links?
> 
> A winbox, eh?
> 
> Well, there are dozens of freeware and shareware (crack available)
> windoze programs to recursively download websites. Sites like
> www.tucows.com, www.nonags.com, www.shareware.com etc list them by the
> score.
> 
> You might need to try a few until you hit on one which doesn't molest
> the '//' in freenet URIs.
> 
> Once you choose a site downloader program, you'd need to point it at:
> - http://localhost:8888/[EMAIL PROTECTED]/TFE//
> - http://localhost:8888/[EMAIL PROTECTED]/YoYo//
> or, point it at whatever site(s) you need, observing the above URL
> syntax. Be sure to disable the timeout (or extend it to an hour or so),
> because these crawler progs are used to web performance.
> 
> One last thing - these crawler progs won't be able to tell one freesite
> from another, since it will perceive all freesites as part of the same
> 'site' at http://localhost:8888/. So take care to set a pattern match
> requirement (unless you want the crawler to suck the whole freenet).
> 
> Cheers
> David
> 
> _______________________________________________
> Support mailing list
> [EMAIL PROTECTED]
> http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support

-- 
Matthew J Toseland - [EMAIL PROTECTED]
Freenet Project Official Codemonkey - http://freenetproject.org/
ICTHUS - Nothing is impossible. Our Boss says so.

Attachment: signature.asc
Description: Digital signature

_______________________________________________
Support mailing list
[EMAIL PROTECTED]
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:[EMAIL PROTECTED]

Reply via email to