Hi Karl,
On 2015-06-04 12:30 AM, Karl Tomlinson wrote:
jma...@mozilla.com writes:
>2) compare-talos is in perfherder
>(https://treeherder.mozilla.org/perf.html#/comparechooser), other instances of
>compare-talos have a warning message at the top indicating you should use
>perfherder. We will deprecate those instances of compare-talos next quarter
>completely.
This is very helpful to present PGO separated and to know that
higher is better for canvasmark, but it is not yet ready to
replacehttp://perf.snarkfest.net/compare-talos/index.html
The treeherder version seems to randomly choose which and how many
of the results to load and so the comparison changes after
reloads of the page.
So this is a bug. :) Could you please file something here:
https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management&component=Perfherder
For best results, include reproduction steps and comparison with the
existing snarkfest interface.
>upcoming work:
>2) continue polishing perfherder graphs, compare-view
Perhaps the above issue is already in this work and you know it
will be addressed by next quarter, but, if not, can we keep the
snarkfest version running please until this is resolved?
Today, the treeherder version is not loading enough results to do a
reasonable comparison, while the snarkfest version doesn't seem to
have the problem and presents results almost instantly.
If I may sneak in a request or two, then a number of results or an
estimate of standard error in the mean would be helpful to know
how to interpret the standard deviations. Also, a way to see
whether higher or lower is better even for those tests without
enough data to detect a statistically significant change would be
a bonus.
These sound like good ideas. Can't remember whether we've filed issues
for them yet. Feel free to have a look at the perfherder bug list and
file a feature request if something isn't open yet:
https://wiki.mozilla.org/Auto-tools/Projects/Perfherder#Bug_Table
I'm keen to find out what is in the "Details" links, but they
currently just ask me to "wait a minute".
Likewise, that sounds like a bug. Clicking on details is supposed to
open up a subtest summary. Here it seems to work fine for me:
https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&originalRevision=b2ad2f9f8e53&newProject=mozilla-central&newRevision=f21fac50a8cc
Will
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
