On 3/4/13 8:15 AM, Jim Mathies wrote:
So to work around this I’ve been putting together some basic perf tests I can use to measure performance using the mochitest framework.
How are you dealing with the fact that mochitest runs on heterogeneous hardware (including VMs and the like last I checked, which could have arbitrarily bad (or good!) performance characteristics depending on what else is happening with the host system)?
Maybe we should consider changing this system so devs can write performance tests that suit their needs that are integrated into our main repo? Basically: 1) rework graphs server to be open ended so that it can accept data from test runs within our normal test frameworks. 2) develop of test module that can be included in tests that allows test writers to post performance data to graph server. 3) come up with a good way to manage the life cycle of active perf tests so graph server doesn’t become polluted. 4) port existing talos tests over to the mochitest framework. 5) drop talos.
This sounds plausible, modulo the inability to port Tp in its current state to a setup that involves the tests living in m-c, as long as the problem above is kept in mind. Basically, reusing something mochitest-like for developer familiarity may make sense, but it would need to be a separate test suite run on completely separate test slaves that are actually set up with performance testing in mind. A separate test suite which is like mochitest is not a problem per se (we have the ipcplugins, chrome, browserchrome, a11y tests already).
So the main win would be making it easier to add new tests in terms of number of actions to be taken (something it seems like we could improve with the current Talos setup too) and easier for developers to add tests because the framework is already similar, right?
-Boris _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
