Hello, I would like to index many files using a mysql database. Before a file should be indexed I would like to check the content with a perl script, which also decides if the file is worth to be indexed.
Before a file will be downloaded I would like to check with a perl code if the link seams to be interesting and make a decision according to the link name. The Web Interface to access the crawlers database should be in PHP. The layout I would like to make like google. If it is possible to implement own ranking modules it would be fine. Currently I use LWP::RobotUA and write the interesting files completely in a mysql database and search the database with LIKE '%$WORD%' which is very slow. Now there are multiple systems available: DBIx::KwIndex, DBIx::Fulltext, DBIx::TextIndex Glimpse, htdig, SIWSH-E, Isearch, WordIndex Could somebody recommend on of this systems for this needs. Many thanks Matthias Jaekle _______________________________________________ Robots mailing list [EMAIL PROTECTED] http://www.mccmedia.com/mailman/listinfo/robots
