Some thoughts on the subject- I would argue against running performance tests inside of mochitest. The main reason is that mochitest has a lot of profile stuff for testing as well as many other tests bundled inside of the same browser session. For a standalone metric unrelated to a user scenario, we could consider performance style tests into mochitest.
In the process of creating Datazilla, we have found endless little quirks in the end to end system how performance works. As time goes on we have continued to push forward with the goal of making a performance system that can detect regressions automatically when the test finishes. For the last few months we have had data going both to Datazilla and graph server and have been refining our assumptions and tools along the way. When graph server is deprecated in the near future, it will be REALLY EASY to add new tests to the collection and reporting system. That doesn't solve the problem of making it easy to add or adjust a test in the test runners (buildbot scripts), but it solves half the problem. Many of the talos tests are old and outdated and while we have tried to find owners for the tests, it has been a failing effort. To that tune, we have disabled some Talos tests which nobody had interest in anymore. If there are tests which people feel are not useful, we should disable those ASAP to reduce our load on our infrastructure and work on creating a test which people care about. -Joel ----- Original Message ----- From: "Ed Morley" <emor...@mozilla.com> To: "Jim Mathies" <jmath...@mozilla.com>, auto-to...@mozilla.com Cc: dev-platform@lists.mozilla.org Sent: Monday, March 4, 2013 8:42:39 AM Subject: Re: proposal: replace talos with inline tests (CCing auto-to...@mozilla.com) jmaher and jhammel will be able to comment more on the talos specifics, but few thoughts off the top of my head: It seems like we're conflating multiple issues here: 1) "[talos] is a separate repo from mc" 2) "[it's a hassle to] test the test to be sure it’s working" 3) "[it's a hassle to get results] populated in graph server" 4) "[we need to] come up with a good way to manage the life cycle of active perf tests so graph server doesn’t become polluted" Switching from the talos harness to mochitest doesn't fix #2 (we still have to test, and I don't see how it magically becomes any easier without extra work - that could have been applied to talos instead) or #3/#4 (orthogonal problem). It also seems like a brute force way of fixing #1 (we could just check talos into mozilla-central). Instead, I think we should be asking: 1) Is the best test framework for performance testing: [a] talos (with improvements), [b] mochitest (with a significant amount of work to make it compatible), or [c] a brand new framework? 2) Regardless of framework used, would checking it into mozilla-central improve dev workflow enough to outweigh the downsides (see bug 787200 for history on that discussion)? 3) Regardless of framework used, how can we make the development/testing/staging cycle less painful? 4) Regardless of framework used, who should be responsible for ensuring we actively prune performance tests that are no longer relevant? Note also that graphs.mozilla.org will be depreciated soon, in favour of datazilla - which afaik is less painful for adding new test suites (eg doesn't need manual database changes); jeads can say more on that front. Best wishes, Ed On 04 March 2013 13:15:56, Jim Mathies wrote: > For metrofx we’ve been working on getting omtc and apzc running in the > browser. One of the things we need to be able to do is run performance tests > that tell us whether or not the work we’re doing is having a positive effect > on perf. We currently don’t have automated tests up and running for metrofx > and talos is even farther off. > > So to work around this I’ve been putting together some basic perf tests I can > use to measure performance using the mochitest framework. I’m wondering if > this might be a useful answer to our perf tests problems long term. > > Putting together talos tests is a real pain. You have to write a new test > using the talos framework (which is a separate repo from mc), test the test > to be sure it’s working, file rel eng bugs on getting it integrated into > talos test runs, populated in graph server, and tested via staging to be sure > everything is working right. Overall the overhead here seems way too high. > > Maybe we should consider changing this system so devs can write performance > tests that suit their needs that are integrated into our main repo? Basically: > > 1) rework graphs server to be open ended so that it can accept data from test > runs within our normal test frameworks. > 2) develop of test module that can be included in tests that allows test > writers to post performance data to graph server. > 3) come up with a good way to manage the life cycle of active perf tests so > graph server doesn’t become polluted. > 4) port existing talos tests over to the mochitest framework. > 5) drop talos. > > Curious what people think of this idea. > > Jim > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
