You may want to use 'wget' spider function, then parse the files with PHP if necessary.


On Thu, 24 Jan 2002 14:52:54 +0100, Mattias Andersson wrote:

>I was making a spider for a simple searchengine and all was well until I
>started testing it on larger sites.
>The problem isn't that it doesn't work, it does, but it is very very slow,
>just handling one connection at the time.
>Basically, it just downloads a page, extract all links from it, then
>downloads those pages, extract their links, until all pages on a site has
>been downloaded and archived.
>Is there any way to do it in several threads so that it handles several
>connections simultaneously?
>If it would dynamically allocate threads that would be even better.
>Mattias Andersson
>Software Developer, humany AB
>Direct: 08-4540701
>Mobile: 0704-526685

PHP General Mailing List (
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]

Reply via email to