"Boris Zbarsky" <bzbar...@mit.edu> wrote in message news:<o7ydnyp6n66okqnmnz2dnuvz_uwdn...@mozilla.org>... > On 3/4/13 8:15 AM, Jim Mathies wrote: > > So to work around this I’ve been putting together some basic perf tests I > > can use to measure performance using the mochitest framework. > > How are you dealing with the fact that mochitest runs on heterogeneous > hardware (including VMs and the like last I checked, which could have > arbitrarily bad (or good!) performance characteristics depending on what > else is happening with the host system)?
That sounds like a rel eng problem that could be solved. I don’t know our enough about our test slaves to say for sure. > This sounds plausible, modulo the inability to port Tp in its current > state to a setup that involves the tests living in m-c, as long as the > problem above is kept in mind. Basically, reusing something > mochitest-like for developer familiarity may make sense, but it would > need to be a separate test suite run on completely separate test slaves > that are actually set up with performance testing in mind. A separate > test suite which is like mochitest is not a problem per se (we have the > ipcplugins, chrome, browserchrome, a11y tests already). That's fine, I'm not married to mochitest, but something similar using the similar run characteristics would be best. > So the main win would be making it easier to add new tests in terms of > number of actions to be taken (something it seems like we could improve > with the current Talos setup too) and easier for developers to add tests > because the framework is already similar, right? > > -Boris Yes, basically - 1) something checked into mc anyone can easily author or run (for tracking down regressions) without having to checkout a separate repo, or setup and run a custom perf test framework. 2) performance tests that generate data that spits out to the console on local runs or could be posted to a graphs server in automation. 3) no releng overhead for setup of new perf tests. something that is built into the test framework / infrastructure we set up. Jim _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
