Hello!
I was making a spider for a simple searchengine and all was well until I
started testing it on larger sites.
The problem isn't that it doesn't work, it does, but it is very very slow,
just handling one connection at the time.
Basically, it just downloads a page, extract all links from it, then
downloads those pages, extract their links, until all pages on a site has
been downloaded and archived.
Is there any way to do it in several threads so that it handles several
connections simultaneously?
If it would dynamically allocate threads that would be even better.
Regards,
Mattias Andersson
Software Developer, humany AB
Direct: 08-4540701
Mobile: 0704-526685
