Check out PHPdig;
http://www.phpdig.net/
Warren
-Original Message-
From: Jamie Alessio [mailto:[EMAIL PROTECTED]
Sent: Thursday, February 17, 2005 9:22 AM
To: John Cage
Cc: php-general@lists.php.net
Subject: [PHP] Re: Crawlers (was parsing large files - PHP or Perl)
Is there anyone on this list who has written fast and decent
crawlers in PHP who would be willing to share their experiences?
My first inclination would be to use an existing crawler to grab the
pages and store all the files locally (even if only temporarily). Then,
you can use PHP to do whatever type of processing you want on those
files and can even have PHP crawl deeper based on links in those files
if necessary. I'd have a hard time coming up with a reason to think I
would implement a better web crawler on my own than is already available
from other projects that focus on that. What about existing search
systems like:
Nutch - http://www.nutch.org
mnoGoSearch - http://mnogosearch.org/
htdig - http://www.htdig.org/
or maybe even a wget -r - http://www.gnu.org/software/wget/wget.html
(I'm sure I missed a bunch of great options)
Just an idea - I'd also like to hear if someone has written nice
crawling code in PHP.
- Jamie
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
