On Mon, 26 Apr 2010 11:40:55 +0100, Jack Stringer wrote: >>>> since checking links to Wikipedia seems to be a legitimate task for >>>> Xenu, shouldn't someone contact them and as for the removal of the >>>> robots.txt exclusion?. Or is there a reason that Xenu and Wikipedia >>>> don't work together smoothly, e.g because of the internal >>> redirects in Wikipedia? >>>> >>>> By the way, >>>> >>>> User-agent: Xenu >>>> Disallow: / >>>> >>>> is also contained in http://de.wikipedia.org/robots.txt. >>>> <http://de.wikipedia.org/robots.txt.> > > >There are a couple of thousand users using Xenu if they all started >sending requests to wikipedia site then the server soon gets bogged down >trying to deliver the pages. Its the same as those people using website >copying software. I have had my photography gallery go very very slow at >times just because someone is trying to hoover up the pictures. > >What would be nice is to find out from wikipedia what changes need to be >made to Xenu so make it nicer to their systems. E.g some sort of delay >when getting pages from wikipedia servers.
Xenu is already "nice", i.e. it makes a HEAD request, not a GET request. My opinion is that the wikipedia software is crappy. The organisation is mostly concentrated on collecting money, enforcing censorship, altering history, and being busy with itself (many of the admins are just very intelligent kids with too much time), instead of delivering a high quality product by running a Continuous Improvement Process. Tilman (holder of a scarlet letter from the wikipedia arb board :-)) http://en.wikipedia.org/wiki/User:Tilman > > >Jack Stringer > > >------------------------------------ > >Yahoo! Groups Links > > >
