There has been some uncertainty about how to react/what to do when you get assigned a performance regression bug. Typically these bugs are noticed by the automated performance tests, and a bug report is filed by someone (often me).
I wouldn't expect you to drop everything instantly and work on the issue until you have figured it out. But I do think it would be easier on you to spend some time on the issue in the first couple of days rather than wait for longer before attacking it. Also, sometimes the bug report is invalid or turns out to be an issue that cannot be reproduced. Rather than spend hours and hours on trying to figure out if there really is a bug, I would advice you to run a quick test: record the time it takes for the test in the working revision, then record the time it takes to run with the revision where problem was first noticed. Run these tests 2-3 times to see how much the normal variance is. If you do not detect a significant change, ask someone else (for example from QA) to try and reproduce the issue. If nobody can reproduce the issue, we should wait at least one day and see if the automatically reported numbers change. We have seen cases where Tinderbox numbers unexpectedly change from day to day (seems to happen on Windows, not so much on the other platforms). In any case, if nobody can reproduce it in a couple of days it is fair to mark it as worksforme. Finally, remember that to get accurate performance numbers you need to run the tests on machines that are not doing anything else. This means that the shared linuxdev machine is not a good platforms to run performance tests. Also, running tests over the network (displaying Chandler on remote desktop), will not give accurate numbers. If the change was large enough, you may be able to detect it even with these unsuitable settings but there are no guarantees. -- Heikki Toivonen
signature.asc
Description: OpenPGP digital signature
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Open Source Applications Foundation "Dev" mailing list http://lists.osafoundation.org/mailman/listinfo/dev
