On Mon, Feb 23, 2015 at 5:59 AM, Andrew Gierth <and...@tao11.riddles.org.uk> wrote: >>>>>> "Tomas" == Tomas Vondra <tomas.von...@2ndquadrant.com> writes: > Tomas> Interesting, but I think Gavin was asking about how much > Tomas> variation was there for each tested case (e.g. query executed on > Tomas> the same code / dataset). And in those cases the padding / > Tomas> alignment won't change, and thus the effects you describe won't > Tomas> influence the results, no? > > My point is exactly the fact that since the result is not affected, this > variation between runs of the same code is of no real relevance to the > question of whether a given change in performance can properly be > attributed to a patch. > > Put it this way: suppose I have a test that when run repeatedly with no > code changes takes 6.10s (s=0.025s), and I apply a patch that changes > that to 6.26s (s=0.025s). Did the patch have an impact on performance? > > Now suppose that instead of applying the patch I insert random amounts > of padding in an unused function and find that my same test now takes a > mean of 6.20s (s=0.058s) when I take the best timing for each padding > size and calculate stats across sizes. Now it looks obvious that the > actual code of the patch probably wasn't responsible for any change... > > The numbers used here aren't theoretical; they are obtained by testing a > single query - "select * from d_flt order by v offset 10000000" where > d_flt contains 5 million float8 values - over 990 times with 33 > different random padding sizes (uniform in 0-32767). Here's a scatter > plot, with 3 runs of each padding size so you can see the repeatability: > http://tinyurl.com/op9qg8a
Neat chart. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
