I have a single site that has 4 main directories that need to be indexed; all together there are about 200,000 documents. 2 of these directories contain documents that don't ever change, and they take up about 70% of the total number of documents. The other 2 directories change daily; there are generally anywhere from 50 to 300 new or changed documents every day. (These documents are Wordperfect and Word documents that have been converted to html nightly as part of a cron job using some custom perl scripts and a convertion tool called wp2html). I need to update the changing directories nightly so I can search on these new and changed documents.
When I initially ran index, the database was created just fine and I was able to search the documents that I needed. Then I started running nightly index jobs that took about 30 to 40 minutes to run, but I wasn't seeing any changes to the old documents, and it didn't really look like any new documents were being added either (all of the documents contain last modified dates that I was using to search on). After poking around in the aspseek.conf file I discovered the period command was set to 7d (7 days) and I figured that was my problem, so I lowered this to 6h (6 hours). Now my index is running but it is taking a really long time to run (6 hours so far). Looking at the logs.txt file, it looks like it is indexing everything from scratch (the queued docs count is up to over 100,000 documents).
Is there a way that I can configure AspSeek to only look for updates in the 2 directories that contain changes? Or can I configure searchd to search 2 different databases at the same time when a search request is made?
Or (and this is a more complicated question) can I call index to insert or update a single document at a time? If this works then I can just add this to my conversion script because it already goes through and finds new and changed documents as part of its process.
My goal here is to be able to run these update scripts overnight so that any changes made the previous day are searchable.
Thanks for the advice.
-- Daniell Freed Computer Services Dewitt, Ross, & Stevens S.C. He who fights with monsters might take care lest he thereby become a monster. And if you gaze for long into an abyss, the abyss gazes also into you.
Beyond Good and Evil Friedrich Wilhelm Nietzche
