> -----Original Message----- > From: [email protected] > [mailto:[email protected]] > On Behalf Of Max Pyziur > Sent: 09 November 2009 01:09 > To: [email protected] > Subject: [Mailman-Users] Using robots.txt > > Greetings > > Mailman's email lists are visible search engine spiders from > http://www.somedomain.com/pipermail/emaillistname/etc > > However, /pipermail is an alias of /var/mailman/archives/public/ > > per mailman.conf > > I've tried placing a basic robots.txt file at > /var/mailman/archives/public/ and set permissions to 644. > However, my lists still get spidered. > > Any suggestions on where to place the robots.txt file to > prevent spidering? > > Thanks! > > Max Pyziur > [email protected]
Hello Max, AFAIK your robots.txt file should be in the TOP level directory of your website, so that it is browseable via http://www.somedomain.com/robots.txt . This is the default location for it and 'good' spiders will look for it there. It should contain the allow/deny details for the whole of your website and in your case would look something like this: ---- User-agent: * Disallow: /pipermail/ ---- .. Which informs all browsers to disallow any URL starting (containing?) the phrase "/pipermail/". Give that a whirl and see how it does. Regards, Steff Watkins ======= ------------------------------------------------------ Mailman-Users mailing list [email protected] http://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-users/archive%40jab.org
