Redhat 7.2, <http://www.htdig.org/>ht://Dig 3.2.0b4-20030216 - I have noticed, going back to 3.1.5, that htdig slows down its indexing speed over time. That is, for the first hour or two of indexing over local files access, htdig indexes roughly 1,000 pages per minute (average size 12Kb). From then on it appears to slow down considerably so that a dig which should take circa 5 hours (300,000 pages/60,000/hour) ends up taking 18 hours. Is this an obvious result of the increasing size of the database, or should I be looking for an answer elsewhere (I/O etc.)?
You might want to keep an eye on memory use (via top, or something similar) during the dig. It might be that after the first hour or two, things grow to the point that you start dipping into swap. At that point, disk access can increase significantly and thus slow everything down.
Jim
-------------------------------------------------------
This SF.net email is sponsored by: Etnus, makers of TotalView, The debugger for complex code. Debugging C/C++ programs can leave you feeling lost and disoriented. TotalView can help you find your way. Available on major UNIX and Linux platforms. Try it free. www.etnus.com
_______________________________________________
htdig-general mailing list <[EMAIL PROTECTED]>
To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe
FAQ: http://htdig.sourceforge.net/FAQ.html
