On Tue, Mar 14, 2017 at 3:45 PM, Kevin Grittner <kgri...@gmail.com> wrote: >> On 3/14/17 17:34, Mengxing Liu wrote: >>> There are several alternative benchmarks. Tony suggested that we >>> should use TPC-E and TPC-DS. > > More benchmarks is better, all other things being equal. Keep in > mind that good benchmarking practice with PostgreSQL generally > requires a lot of setup time (so that we're starting from the exact > same conditions for every run), a lot of run time (so that the > effects of vacuuming, bloat, and page splitting all comes into play, > like it would in the real world), and a lot of repetitions of each > run (to account for variation). In particular, on a NUMA machine it > is not at all unusual to see bifurcated
Sorry I didn't finish that sentence. On a NUMA machine It is not at all unusual to see bifurcated results -- with each run coming in very close to one number or a second number, often at about a 50/50 rate, with no numbers falling anywhere else. This seems to be based on where the processes and memory allocations happen to land. Different people have dealt with this in different ways. If you can get five or more runs of a given test, perhaps the best is to throw away the high and low values and average the rest. Other approaches I've seen are to average all, take the median, or take the best result. The worst result of a set of runs is rarely interesting, as it may be due to some periodic maintenance kicking in (perhaps at the OS level and not related to database activity at all). -- Kevin Grittner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
