Hello, since checking links to Wikipedia seems to be a legitimate task for Xenu, shouldn't someone contact them and as for the removal of the robots.txt exclusion?. Or is there a reason that Xenu and Wikipedia don't work together smoothly, e.g because of the internal redirects in Wikipedia?
By the way, User-agent: Xenu Disallow: / is also contained in http://de.wikipedia.org/robots.txt. All the best Thomas ________________________________ Von: [email protected] [mailto:[email protected]] Im Auftrag von Vince Thacker Gesendet: Montag, 12. April 2010 18:48 An: [email protected] Betreff: Re: [xenu-usergroup] link to wikipedia I've raised a similar query before about many Google links that return a 403, especially their RSS feeds. It doesn't seem you can do much about it, as these sites have a policy of forbidding software such as Xenu getting access. Presumably if a page is generating a 403, you know at least that you have a valid link. Vince. ----- Original Message ----- From: "Lars" <[email protected]<mailto:baloo5419%40yahoo.co.uk>> To: <[email protected]<mailto:xenu-usergroup%40yahoogroups.com>> Sent: Monday, April 12, 2010 4:04 PM Subject: [xenu-usergroup] link to wikipedia > > I have quite some links to Wikipedia. > > They give this message. A little annoying > error code: 403 (forbidden request) > > > > > ------------------------------------ > > Yahoo! Groups Links > > > > >
