[htdig] speed of indexing slows down over time

[Greg Lepore]

        Redhat 7.2, <http://www.htdig.org/>ht://Dig 3.2.0b4-20030216 - I 
have noticed, going back to 3.1.5, that htdig slows down its indexing speed 
over time.  That is, for the first hour or two of indexing over local files 
access, htdig indexes roughly 1,000 pages per minute (average size 
12Kb).  From then on it appears to slow down considerably so that a dig 
which should take circa 5 hours (300,000 pages/60,000/hour) ends up taking 
18 hours.  Is this an obvious result of the increasing size of the 
database, or should I be looking for an answer elsewhere (I/O etc.)?
Gregory Lepore
Webmaster, State of Maryland
Supervisor, Archives of Maryland Online
410-260-6425



-------------------------------------------------------
This SF.net email is sponsored by: Etnus, makers of TotalView, The debugger 
for complex code. Debugging C/C++ programs can leave you feeling lost and 
disoriented. TotalView can help you find your way. Available on major UNIX 
and Linux platforms. Try it free. www.etnus.com
_______________________________________________
htdig-general mailing list <[EMAIL PROTECTED]>
To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe
FAQ: http://htdig.sourceforge.net/FAQ.html