The JS tests can / do run on device though? Having extra testing is always a good thing, but I think the point Gareth was making is that we should be looking to avoid having competing suites which is often duplicating effort.
The developers are using the js tests, they have the same capability (afaik) as the python suite, it is part of our workflow (and a documented requirement) that we test functionality we are landing, that will as far as I know always be done in the js suite. Therefore is seems that test effort should go into working on the test suite that is part of the developers workflow, its going to mean the test suite turns red less and there is far less baggage involved in learning the dev workflow. Personally I would have been happy (and tbh probably preferred) using the existing python infrastructure, however as far as I can tell that ship has sailed and 'make test-integration' is now burned into gaia devs brains and theres huge value in having a single canonical test suite (as if we dont have enough problems with multiple build configurations). The offer to help is definitely very much appreciated, would be awesome to get a common workflow agreed on here. On 19 November 2013 22:58, Naoki Hirata <[email protected]> wrote: > They are valuable. Personally I see them as another tier to testing. > > Js catches changes upon checkin; mochi tests capture changes during build > time and python tests capture changes after build on real devices. This > is before the smoke tests and manual testing so we can better our time on > the manual testing. Just because it doesn't happen in the JS level or > Mochi test level, it could happen on the actual device because of driver > issues. > > I think it would be good that people be aware of what we're trying to > capture in each of the automation and why they are set up. :) > > Note: that I state changes not bugs. Because a change in result of the > test doesn't necessarily mean that it's a bug in the change in code; it > could mean that it's the test that needs changing. > > Regards, > Naoki > > On Nov 19, 2013, at 2:50 PM, Zac Campbell <[email protected]> wrote: > > > Well geez i was just offering, kindly I thought, to have QA write some > additional test coverage, especially against real device hardware. I > thought it would be valuable. > > > > But I can read between the lines; you want Python and QA's tests to sod > off. > > > > > > > > ----- Original Message ----- > > From: Gareth Aye <[email protected]> > > To: Zac Campbell <[email protected]> > > Cc: Jonas Sicking <[email protected]>, [email protected], > dev-b2g <[email protected]> > > Sent: Tue, 19 Nov 2013 14:28:43 -0800 (PST) > > Subject: Re: [b2g] Let QA write your Gaia/Marionette UI tests! > > > > Ok well that's all well and good, but I think we should be careful about > > how we do this. Will there be separate pull requests to Gaia? I posit > that > > the correct way to develop is to review and land test code alongside > > feature/bug code. Let's prove this to ourselves by exploring other, wrong > > ways in which we can do this. > > > > Case 1: Suppose we review/land our tests before the feature/bug code. > > > > - This breaks the build. > > - It's possible that we learn things while writing the feature/bug code > > about the constraints of the problem and that the feature changes. In > this > > case, our tests have gotten old and crufty before they ever passed! > > - It makes our generous reviewers take two steps out of their busy days > > instead of one. > > > > Case 2: Suppose we review/land our tests after the feature/bug code. > > > > - We check untested code into source control. Do we even know if it > works? > > - If the feature/bug code gets backed out and the test stays, then we > break > > the build. > > - We waste our reviewer's time again making them review two patches. > > > > Our proof is finished since the only options are to land our test code > > before, alongside, or after our bug code. Therefore, if we're going to > have > > people other than feature/bug code writers developing automated test > cases, > > then everyone who's working on the bug must work together to submit a > > single patch and/or pull request to Gaia. This workflow isn't what I've > > observed so far with the Python test code, and I hope that we keep to it > > for the future health of our project. > > > > Personally this seems like a lot of hassle to me (which is why I prefer > to > > write my own tests), but I am happy for others to do this so long as they > > do it thoughtfully (in a way that doesn't waste reviewer time, stop good > > tests from being written, and break the build). > > > > > > > > On Tue, Nov 19, 2013 at 3:54 AM, Zac Campbell <[email protected]> > wrote: > > > >> Well I was trying to offer you a QA/automation team to do that for you! > >> > >> but I am interested in new test cases for new functionality coming up > that > >> QA haven't been let in on yet or if there's coverage that your team > needs > >> for on-device testing to use hardware that desktopb2g doesn't have > access > >> to. We can cover some space that Travis cannot. > >> > >> Julien has requested some for Messaging app that we did and Rudy also > >> asked for a keyboard/TBPL test which we did last week. > >> > >> > >> > >> > >> > >> > >> > >> On 18/11/13 16:58, Gareth Aye wrote: > >> > >> I will let you, but only if you use the js tooling so that I can review > >> and maintain the tests :) > >> > >> > >> On Sat, Nov 16, 2013 at 3:47 PM, Jonas Sicking <[email protected]> > wrote: > >> > >>> On Nov 14, 2013 3:57 AM, "Zac Campbell" <[email protected]> wrote: > >>>> > >>>> Are you tired? is the cat bothering you for its food? Forgotten to > shave > >>> for the last 6 days? Don't have time to write your UI tests? > >>>> > >>>> Then let QA do it for you! > >>> > >>> Hahaha, I love it! Zac++ > >>> > >>> / Jonas > >>> _______________________________________________ > >>> dev-b2g mailing list > >>> [email protected] > >>> https://lists.mozilla.org/listinfo/dev-b2g > >>> > >> > >> > >> > >> -- > >> Best, > >> Gareth > >> > >> > >> > > > > > > -- > > Best, > > Gareth > > > > _______________________________________________ > > dev-b2g mailing list > > [email protected] > > https://lists.mozilla.org/listinfo/dev-b2g > > _______________________________________________ > dev-b2g mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-b2g > _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
