Hi all, I was wondering if anybody had some advice for me. I'm searching a number of domains that I don't control, and quite a few of them are built with PHP scripts that put a session ID or some other gibberish in the URL. Which seems to change periodically. So, three days later, htdig is still indexing some phpBB forum over and over again, because the URL keeps changing...
Short of telling htdig not to index anything that has 'php?' in the URL, is there anything I can do to fix this? I've been banning htdig from indexing things with characteristic substrings, but my list is getting longer and longer, and I'm getting frustrated with having to run every dig twice (once to see what site is making htdig go around in circles this time, and once to block the troublesome pages). Running a search engine has made me really detest PHP... all the sites I have problems with seem to be PHP-based. -Rhonda -- www.write-on.indy || www.write-on.org \/ http://history.ubcengineers.ca/ Discuss the art and craft of writing /\ UBC Engineers History Project That's the problem with world domination. Nobody is willing to wait for it anymore, work slowly towards it, drink more and enjoy the ride more. ------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. Does SourceForge.net help you be more productive? Does it help you create better code? SHARE THE LOVE, and help us help YOU! Click Here: http://sourceforge.net/donate/ _______________________________________________ ht://Dig general mailing list: <[EMAIL PROTECTED]> ht://Dig FAQ: http://htdig.sourceforge.net/FAQ.html List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-general
