Thanks Gilles, I did over look that user_agent attribute and have added that.
Any chances on speeding up indexing? I had to interupt indexing because of the slowness. I can't even index 30k of pages in a 24 hour period. That hurts. Got to have some speed when indexing. Regards, Andy On Fri, 14 Nov 2003, Gilles Detillieux wrote: > According to Andy Lewis: > > Look like the robots.txt file isn't being parsed properly. > > > > I've used the > > <http://www.jumboclassifieds.com/~alewis/attrs.html#robotstxt_name> > > robotstxt_name tag and added the same name to my robots.txt file and I > > still see the > > default htdig name when indexing. > > > > Any ideas? Running the lastest beta. Downloaded today. > > It seems to me you're confusing the robotstxt_name attribute with > the user_agent attribute. If by "I still see the default htdig name" > you mean that's what's showing up in the access_log, then you want to > change user_agent. > > See http://www.htdig.org/dev/htdig-3.2/attrs.html#user_agent > > There is a bug in 3.2.0b5 in that it doesn't correctly handle an empty > Disallow directive, but that doesn't seem to be the issue here. The fix > for this latter bug is at > > ftp://ftp.ccsf.org/htdig-patches/3.2.0b5/robots.0 > > -- > Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]> > Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/ > Dept. Physiology, U. of Manitoba Winnipeg, MB R3E 3J7 (Canada) > ------------------------------------------------------- This SF. Net email is sponsored by: GoToMyPC GoToMyPC is the fast, easy and secure way to access your computer from any Web browser or wireless device. Click here to Try it Free! https://www.gotomypc.com/tr/OSDN/AW/Q4_2003/t/g22lp?Target=mm/g22lp.tmpl _______________________________________________ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
