On Tue, 17 Apr 2007, Heikki Toivonen wrote:
1. Are regression bugs usable to you?
See my replies to you yesterday. If the regression is real and you've talked
to me first, then yes, the bug is real and usable.
2. Are trend graphs usable to you?
http://builds.osafoundation.org/perf_data/trends.html
Sort of, they seem to only make sense in the long run.
3. Are the tables from which trend graphs are drawn from usable to you?
For example
http://builds.osafoundation.org/perf_data/detail_20070318_20070417.html
No. The numbers are all over the map.
4. Are the full results for today usable to you? For example
http://builds.osafoundation.org/perf_data/detail_20070417.html
No. The numbers are all over the map.
5. Are the daily graphs usable to you? For example
http://builds.osafoundation.org/perf_data/detail_20070417.html
No.
6. Are historical daily reports usable to you? For example
http://builds.osafoundation.org/perf_data/detail_20070416.html
No, the interval is too short and the numbers are all over the map.
7. Are the deltas and std.dev usable to you on
http://builds.osafoundation.org/perf_data/tbox.html
No. The numbers are all over the map, even when comparing a version against
itself.
8. If you would like to change the colors or format of
http://builds.osafoundation.org/perf_data/tbox.html please list the
changes here:
No change.
9. Is rt.py -p usable to you?
No, the profiles that come out don't load in hotshot.
10. Is rt.py -t usable to you?
Dunno, I still use do_tests.sh and RunPython directly.
I saw no need to move to rt.py. Maybe there is one, I just don't know about
it.
11. Is rt.py -P usable to you?
Err, maybe my answer for 'rt -p' is the one applicable here.
I guess, I don't know what 'rt -p' does. Run the perf tests maybe ?
I never do, their numbers only make sense over a week or so to get a trend.
Not for a punctual timing. For that, I've only seen direct instrumenting to be
reliable.
12. Is rt.py --repeat usable to you?
Didn't know it existed. See my -P answer.
13. Any other things you can think of that either add value or are
irrelevant or you would like changed:
The perf tboxes are too random to be useful for anything beyond detecting a
trend over a week or so of runs.
Andi..
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
