Alec Flett wrote: > For instance, say I broke something yesterday at 5pm, and the delta > shows up at 7pm and displays until 9pm. Nobody is going to notice that. > We'll all come in in the next morning and see that some of the > performance numbers are red... but aren't there always SOME numbers red? > Only someone such as yourself, who knows the tests well enough to know > what's normally red and whats not, will notice that something is amiss.
I actually tend to take a look at the perf graphs as the last thing each evening, so I tend to notice them (looking at the day graph). Obviously, I'd rather be able to take a look next morning but that isn't possible until we get the graph extending further back. If the perf jump happened on all platforms with the same checkin, then it seems real even if there is some random noise. Changes on a single platform are harder, but if the jump levels off around the new value, then it's usually real. The Mac has the most fluctuation in values, so it is the least reliable of the perf boxes we have. There are also some tests that have more fluctuation than others. Generally, the less time a test takes the more it is susceptible to fluctuation. Some of the worst tests fluctuate up to 30% on the Mac, but when you plot a line through it is pretty level. -- Heikki Toivonen
signature.asc
Description: OpenPGP digital signature
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Open Source Applications Foundation "Dev" mailing list http://lists.osafoundation.org/mailman/listinfo/dev
