You may want to use 'wget' spider function, then parse the files with PHP if necessary.
On Thu, 24 Jan 2002 14:52:54 +0100, Mattias Andersson wrote:
>I was making a spider for a simple searchengine and all was well until I
>started testing it on larger sites.
>The problem isn't that it doesn't work, it does, but it is very very slow,
>just handling one connection at the time.
>Basically, it just downloads a page, extract all links from it, then
>downloads those pages, extract their links, until all pages on a site has
>been downloaded and archived.
>Is there any way to do it in several threads so that it handles several
>If it would dynamically allocate threads that would be even better.
>Software Developer, humany AB
PHP General Mailing List (http://www.php.net/)
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]