On Sat, 28 Aug 2004 [EMAIL PROTECTED] wrote:
I've done a few more trial runs and it's still conking out midway through indexing the site.
The funny thing is, the host not found/no server running stuff stops after it's managed to index 16,000+ pages...what suddenly makes it forget its DNS info.
Any advice?
Did you try setting ignore_dead_servers to false as I suggested earlier? If you have it set to true, or are using the default, the server only gets one chance; if htdig decides it is unreachable or not running, it is flagged as dead and remains that way for the rest of the dig. So if there is one glitch halfway through, that would be enough to prevent you from indexing the rest of the site. If you set the attribute to false, it will attempt to make a connection for every URL on its list, regardless of whether or not the server appears to be accessible and responding.
If you already have the attribute set to false and are still seeing htdig fail consistently through the latter part of the run, then the most likely cause is that there is in fact a problem related to your network or the server being indexed. With ignore_dead_servers set to false, htdig should be making an honest attempt to connect to the server for each and every URL that is in its list and within specified limits. In this case, one possibly useful experiment would be to use a browser on the indexing machine to request URLs from the server at the same time that htdig is
reporting problems with the server.
If you are not already doing so, you might also want to try setting the server_wait_time attribute.
http://www.htdig.org/attrs.html#server_wait_time
This might help out if there is some sort of load related problem involved.
Jim
------------------------------------------------------- This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools! Get your free copy of BEA WebLogic Workshop 8.1 today. http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click _______________________________________________ ht://Dig general mailing list: <[EMAIL PROTECTED]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general
