Author: pebender
Date: Sun Apr 19 11:29:52 2009
New Revision: 4635
Modified:
trunk/gar-minimyth/html/minimyth/robots.txt
Log:
- Updated www.minimyth.org robots.txt file.
Modified: trunk/gar-minimyth/html/minimyth/robots.txt
==============================================================================
--- trunk/gar-minimyth/html/minimyth/robots.txt (original)
+++ trunk/gar-minimyth/html/minimyth/robots.txt Sun Apr 19 11:29:52 2009
@@ -1,6 +1,11 @@
User-agent: *
+Disallow: /css/
Disallow: /error/
-
+Disallow: /image/
# Don't crawl the download directory as it changes to often.
-User-agent: *
Disallow: /download/
+# Don't crawl non-forum parts of the forum.
+Disallow: /forum/faq.php
+Disallow: /forum/groupcp.php
+Disallow: /forum/login.php
+Disallow: /forum/privmsg.php
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"minimyth-commits" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/minimyth-commits?hl=en
-~----------~----~----~----~------~----~------~--~---
