Look into Pavuk, webBase, wget, or Larbin. They might satisfy most of
the requirements (though not all) mentioned here.
Hope it helps,
Krishna Jha
Bhasha Inc
PS: actually, this is a repeat of an earlier posting; Nick - is it time
for a InternetRobots FAQ here ?
Jim MacDiarmid wrote:
Is there anything like this that would run on a Windows 98 or NT platform?
Jim MacDiarmid, Senior Software Engineer
PACEL Corp.
8870 Rixlew Lane
Manassas, VA 20109
(703) 257-4759
FAX: (703) 361-6706
www.pacel.com
-Original Message-
From: Simon Wilkinson [SMTP:[EMAIL PROTECTED]]
Sent: Sunday, January 14, 2001 4:37 PM
To: [EMAIL PROTECTED]
Subject: Re: Looking for a gatherer.
I am looking for a spider/gatherer with the following characteristics:
* Enables the control of the crawling process by URL
substring/regexp
and HTML context of the link.
* Enables the control of the gathering (i.e. saving) processes by
URL
substring/regexp, MIME type, other header information and ideally by
some
predicates on the HTML source.
* Some way to save page/document metadata, ideally in a database.
* Freeware, shareware or otherwise inexpensive would be nice.
You might like to take a look at Harvest-NG, which is free software.
(http://webharvest.sourceforge.net/ng) It will allow all of what you
detail above. It saves the metadata in a Perl DBM database - some work
has been done, but not completed, on working with the DBI interface
to a remote database. You may find that some knowledge of Perl is helpful
in adapting it exactly to your needs (much use is made of Perl regular
expressions in the pattern matching, for instance).
Cheers,
Simon.
