Thomi, Just want to give some quick feedback to your request.
On Tue, Nov 19, 2013 at 2:28 PM, Thomi Richards <[email protected]> wrote: > Hi CI team, > > > The mir team would like to run the glmark2 benchmarks on a native mir server > across Desktop, N4 and N10 devices at the MP stage, and block merges on a > significant drop in FPS performance. > > > We already have a glmark2 version that runs against a native mir server. The > parts that are missing includes: > > Infrastructure to run that test (jenkins and whatnot). > > Policy around exactly when to fail the test, when to warn about a drop in > performance, and when to pass the test. > > A way to convert the data from glmark2 into something the CI team can report > on. > > A place to show the graphs of performance over time. > > > So, addressing these in reverse order (why not): > > 4: We already have performance graphs for mir, so I suspect this part will > be easy. I'll assume this can be accomplished under a glmark ci specific page like this one: http://reports.qa.ubuntu.com/graphics/openarena/ > 3: This needs a small amount of python code, I suggest we emit a subunit > result stream here - we could use this as a testbed for subunit. I imagine > that this script would be done by someone in the QA team? Is the CI team > ready to read subunit result streams? I'm more than happy to walk someone > through what needs to happen here (the code is trivially easy)... I agree on using subunit as mentioned here: https://wiki.ubuntu.com/CI/AddingTests (yeah! a new wiki page) > 2: I suggest we get the tests running for a few weeks, graph the results, > and decide the policy based on that data. We're going to be blocking merge > proposals from this, so we need to make sure we get it right. I expect that > the policy would be decided upon by a mixture of the mir and QA teams. The > current idea is that < 5% drop would result in a warning, and >5% would > result in a fail. It would be nice if we could tweak this policy easily > without having to go through the CI team... but that's a minor detail. This may not be so easy. Failing an MP because performance drops below X% is a pass/fail action. Providing a warning when it's not quite so bad is not. We'll need to iterate on how do make this visible to developers. > 1: Chris Gagnon is already working on a jenkins job that runs the mir demos > and integration tests, but I think this should be a separate job, for a > couple of reasons: > > First, we're likely to want to run these performance tests in several places > (MP CI runs to begin with, perhaps on a daily basis on top of the distro > images as well?). > > Second, I think it's worth keeping the concerns of "have we broken the mir > demos or integration tests by altering API without updating them" and the > concern of "have we regressed performance in this release" separate. This should work. The current cupstream2distro-config tools support adding pass/fail child jobs to a project configuration. And this type of test should migrate into the CI Airline world. > So, I don't think there's anything too controversial in here - Can we get a > plan together to figure out who's going to do what, so we can get the mir > team hooked up? I opened a bug to track this request: https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1252933 > Cheers, > > -- > Thomi Richards > [email protected] > > -- > Mailing list: https://launchpad.net/~canonical-ci-engineering > Post to : [email protected] > Unsubscribe : https://launchpad.net/~canonical-ci-engineering > More help : https://help.launchpad.net/ListHelp > -- Francis Ginther Canonical - Ubuntu Engineering - Continuous Integration Team -- Mailing list: https://launchpad.net/~canonical-ci-engineering Post to : [email protected] Unsubscribe : https://launchpad.net/~canonical-ci-engineering More help : https://help.launchpad.net/ListHelp
