On 8 February 2011 21:54, Robert Collins <[email protected]> wrote: > On Tue, Feb 8, 2011 at 10:22 PM, Julian Edwards > <[email protected]> wrote: >> On Tuesday 08 February 2011 02:39:02 Robert Collins wrote: >>> Mid end desktops these days are 8 way machines that should be able to >>> easily do a full test run in < 60 minutes. >> >> Good luck with that. I've parallelised test runs using virtualisation before >> and you get so much I/O contention that the overall increase in speed is only >> about 10%. >> >> (Using two simultaneous runs on my quad-core Phenom) > > I'm certainly aware that there are a bunch of of issues - and previous > experiments here. One of the fairly early steps has to be generating a > load profile for each test process - (python, librarian, appserver, > postgresql etc). We need to then figure out answers to get enough > capacity to scale. For instance, IO load can perhaps be largely > addressed via running the test DB's in a ramdisk. I remember prior > attempts with ramdisks having little impact - but that may can easily > be accounted for if we consider that in a single threaded > configuration we are not bottlenecked on IO: eliminating a > non-bottleneck won't have a dramatic effect. In fact if we're > sufficiently far below the IO thresholds in a single threaded test > config, its plausible we wouldn't see any improvement. > > As a datapoint, bzr, which has parallelised its tests for some years > now, scales nearly linearly - and its very IO heavy, writing GB's of > data across an entire test run. If you hit the IO ceiling on a > machine, it stops improving rapidly ;)
Yes, big thing to do with bzr is to put TMPDIR onto a tmpfs, which will both give you a constant-factor improvement, and also much higher scaling headroom. bzr writes a lot of data, but also deletes it soon afterwards so it's ideal for a ramdisk. Launchpad probably won't get quite that behaviour, but it might get close. Martin _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
