According to Peter Choe: > actually, i read your response and was at a loss as to what you were > talking about. > > i am not really familiar with network setup and dns things. > > but to answer your questions now, htdig is on the same server as the > webserver i am trying to index, and it is just running a webserver.
If htdig is on the same machine as the web server, you should probably use local_urls to speed things up and make them more efficient. See http://www.htdig.org/attrs.html#local_urls > what i am not sure is when you ask me about the name server? are you > referring to the DNS? i don't know how it is set up. Yes, I mean DNS. Whenever htdig sees a host name like foo.bar.com and it wants to open a connection to it, it does a gethostbyname() call to lookup the IP address for that host name via DNS. The system decides which DNS name server to use based on the nameserver record(s) in /etc/resolv.conf. If the IP address for the nameserver is that of a system on your local network, and it's not overloaded, you should be fine with that. However, the sporadic Unknown host errors suggest that the DNS requests are timing out, so htdig can't look up the names consistently. Given that gethostbyname() doesn't do any caching of looked up names, and given that htdig will tend to ask for the same host names over and over, you really want a quick DNS server close by (ideally on the same machine as you're running htdig). As I said last week, many OS distributions give you a pre-packaged set of config files for a caching-only DNS server, so you just have to install it, make sure it's running, and point your /etc/resolv.conf to it. If you need more detailed instructions, it may help to know which OS version and distribution you're running, and what name server packages are available for it. E.g., all versions of Red Hat Linux since at least 4.2 have had a caching-nameserver RPM package which you install along with the "bind" package (the DNS nameserver itself), to get a caching-only configuration. > At 12:41 PM 5/29/2002, Gilles Detillieux wrote: > >According to Greg Burnham: > > > > i am trying to index my webserver with rundig. it is going fine > > until it > > > > comes across an Unknow host: foo.bar.com host not found. > > > > > > > > but htdig was indexing previous pages from that server. everytime i run > > > > rundig to index the site, it seems to stop at different places. > > > > > > > > i am totally at a loss why this is happening. anyone have > > suggestions for me? > > > > > > Have you tried setting server_wait_time in the conf file? Maybe your > > > server is being overloaded and stops responding. Is there any > > > indication in the servers logs? > > > > > > http://htdig.org/attrs.html#server_wait_time > > > >Yes, that's certainly worth a try. > > > >We also never got any feedback from you last week about the possibility of > >the problem being caused by an overloaded name server, or a connection > >that's too slow between your system and its name server. Either of > >these problems may or may not be helped by server_wait_time, depending > >on whether htdig or other activity is the major cause of the slow name > >server response. > > > >If htdig is having problems reaching the name server, that would certainly > >cause those "Unknown host" errors. I mentioned that last week, so I'm not > >sure why you still say you're "totally at a loss why this is happening." > >What is the name server for your htdig system? Is it on your local net, > >or on a remote network? Is it shared by lots of other systems, and > >perhaps overloaded? Is there any possibility of using a closer and/or > >less busy name server for your system, or setting up a caching-only > >DNS server? -- Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/ Dept. Physiology, U. of Manitoba Winnipeg, MB R3E 3J7 (Canada) _______________________________________________________________ Don't miss the 2002 Sprint PCS Application Developer's Conference August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
