Tilman Hausherr wrote: > On Mon, 26 Apr 2010 11:40:55 +0100, Jack Stringer wrote: > >>>>> since checking links to Wikipedia seems to be a legitimate task >>>>> for Xenu, shouldn't someone contact them and as for the removal >>>>> of the robots.txt exclusion?. Or is there a reason that Xenu and >>>>> Wikipedia don't work together smoothly, e.g because of the >>>>> internal >>>> redirects in Wikipedia? >>>>> >>>>> By the way, >>>>> >>>>> User-agent: Xenu >>>>> Disallow: / >>>>> >>>>> is also contained in http://de.wikipedia.org/robots.txt. >>>>> <http://de.wikipedia.org/robots.txt.> >> >> >> There are a couple of thousand users using Xenu if they all started >> sending requests to wikipedia site then the server soon gets bogged >> down trying to deliver the pages. Its the same as those people using >> website copying software. I have had my photography gallery go very >> very slow at times just because someone is trying to hoover up the >> pictures. >> >> What would be nice is to find out from wikipedia what changes need >> to be made to Xenu so make it nicer to their systems. E.g some sort >> of delay when getting pages from wikipedia servers. > > Xenu is already "nice", i.e. it makes a HEAD request, not a GET > request. My opinion is that the wikipedia software is crappy. The > organisation is mostly concentrated on collecting money, enforcing > censorship, altering history, and being busy with itself (many of the > admins are just very intelligent kids with too much time), instead of > delivering a high quality product by running a Continuous Improvement > Process. > > Tilman (holder of a scarlet letter from the wikipedia arb board :-)) > http://en.wikipedia.org/wiki/User:Tilman >
There are plenty of old admins, I can assure you :-) The software is probably rough - it *is* still a charity, and due to the mindless antics of loads of juniville vandals, it needs a large team of vandal fighters (not just admins - there's only 1000 regular ones) to keep the pages more or less intact - English Wikipedia has around 150-200 pages change per minute, and around 10% of those have to be reverted - so the servers are already very busy, and I think allowing Xenu in will grind it to a halt - If the Dutch mirrors go down, and I have to connect direct (from UK) to the USA servers, then it can take 30 seconds plus for a medium page to load. Ron Jones Process Safety & Development Specialist Don't repeat history, unreported chemical lab/plant near misses at http://www.crhf.org.uk Only two things are certain: The universe and human stupidity; and I'm not certain about the universe. ~ Albert Einstein
