On Mon, 24 Jul 2006, Mark Kirkwood wrote: > Tom Lane wrote: > > Mark Kirkwood <[EMAIL PROTECTED]> writes: > >> Scale factor 10 produces an accounts table of about 130 Mb. Given that > >> most HW these days has at least 1G of ram, this probably means not much > >> retrieval IO is tested (only checkpoint and wal fsync). Do we want to > >> try 100 or even 200? (or recommend scale factor such that size > ram)? > > > > That gets into a different set of questions, which is what we want the > > buildfarm turnaround time to be like. The faster members today produce > > a result within 10-15 minutes of pulling their CVS snaps, and I'd be > > seriously unhappy if that changed to an hour or three. Maybe we need to > > divorce compile/regression tests from performance tests? > > > > > > Right - this leads to further questions like, what the performance > testing on the buildfarms is actually for. If it is mainly to catch > regressions introduced by any new code, then scale factor 10 (i.e > essentially in memory testing) may in fact be the best way to show this up.
It introduces a problem though. Not all machines stay the same over time. A machine may by upgraded, a machine may be getting backed up or may in some other way be utilised during a performance test. This would skew the stats for that machine. It may confuse people more than help them... At the very least, the performance figures would need to be accompanied by details of what other processes were running and what resources they were chewing during the test. This is what was nice about the OSDL approach. Each test was preceeded by an automatic reinstall of the OS and the machines were specifically for testing. The tester had complete control. We could perhaps mimic some of that using virtualisation tools which control access to system resources but it wont work on all platforms. The problem is that it probably introduces a new variable, in that I'm not sure that virtualisation software can absolutely limit CPU resources a particular container has. That is, you might not be able to get reproducible runs with the same code. :( Just some thoughts. Thanks, Gavin ---------------------------(end of broadcast)--------------------------- TIP 6: explain analyze is your friend