These places that spider your site, do they just hit the home page or also hit subpages?
I've had my email addresses encoded for some time now, but the SPA* is getting a lot worse at the one
email that I only use on the webpages...
All pages of all depths and all directories UNLESS one of two conditions are found.
1. Robots.txt file that tells the web spiders what directories are OK or not OK 2. Directories where the Unix file permissions are set to not allow reading
Example: Try to see http://www.expita.com/cgi-bin/ whereas this one is open http://www.expita.com/images/
Robots.txt tutorial http://www.searchengineworld.com/robots/robots_tutorial.htm
-- Gerry Boyd ============= PCWorks Mailing List ================= Don't see your post? Check our posting guidelines & make sure you've followed proper posting procedures, http://pcworkers.com/rules.htm Contact list owner <[EMAIL PROTECTED]> Unsubscribing and other changes: http://pcworkers.com =====================================================
