To be clear, you mean manually but not locally, right? I am talking about using automation prior to landing on the trees consumed by downstream, like is done with gecko+Firefox with inbound trees, etc. Things like try servers.
Nobody should have to run them locally and our learning there is that locally run test results are not to be trusted. On Tue, Oct 6, 2015, 11:01 AM Dietrich Ayala <[email protected]> wrote: > Yes, absolutely. It's standard practice to run all of Firefox's > performance tests against Gecko landings. It should not be any different > for FxOS, IMO. > > I've always seen unhiding of all types of Gaia tests on mozilla-central as > a precursor for solving our cross-tree woes. Until that happens, Gaia will > continue to get regressed by Gecko changes, causing noise and churn in the > Gaia dev cycle. > > > > On Tue, Oct 6, 2015 at 10:52 AM Etienne Segonzac <[email protected]> > wrote: > >> >> On Tue, Oct 6, 2015 at 12:47 AM, Eli Perelman <[email protected]> >> wrote: >> >>> If by try runs you mean automated performance testing when opening a PR, >>> then no. Right now the best way to ensure performance is up-to-snuff with >>> your patch is to run Raptor during development. With Raptor installed, use >>> a performance profile on the device by using the same flags as `make >>> raptor`, then testing an app is as easy as: >>> >>> raptor test coldlaunch --app clock --runs 10 >>> raptor test coldlaunch --app communications --entry-point dialer --runs >>> 20 >>> >>> During development use a small number of runs for a tighter feedback >>> loop. Before committing/landing, use many runs to ensure a better >>> statistical guarantee. See the Raptor docs for details on getting started >>> [1]. If you need any help getting up and running with Raptor, Rob Wood and >>> myself would be happy to lend a hand, just ping us [2]. >>> >> >> >> Fabrice, Dietrich, does your "strong stance" on performance includes >> having gecko developers run raptor tests manually before landing too? >> Because without significant infrastructure/tooling investment this is >> just big talk... >> >>
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
