According to John Dispirito:
> I'm currently attempting to spider a site which uses .asp files, and it
> seems like
> the spider isn't working properly. I know its a fairly big site, and most of
> the time the output lists
> maybe at the most 20 -30 spidered entries from the site, and then it just
> stops...
I'd suggest running htdig -i -vvv and looking at the output to see if it's
seeing links to all the pages it should, and if so, why it would be rejecting
many of them. It could be a problem with your exclude_urls or limit_urls_to
settings. Note also that htdig only follows HTML links, not JavaScript.
--
Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]>
Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba Phone: (204)789-3766
Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
