Hi All, As was discussed in the convergence meeting wednesday, we are severely lacking tests.
So I asked Bobby Holley to write some tests for the install/update API that we have had quite a few problems with lately. Both regarding regressions, and regarding various edge and error cases not working. Unfortunately he very quickly ran into the problem that basically the current workflow is so hard to use that it's almost essentially broken. Which is likely why we have essentially no mochitests. The flow when developing a mochitest on desktop is something like the following: 1. Write test 2. Run Firefox such that it opens the single mochitest test that is being developed 3. See test failures 4. Edit test 5. Reload test in Firefox (which is still open since step 2) 6. Go back to step 3. This means that the turnaround time after having edited a test is basically 0. Reloading is instant except in the rare cases when the test takes a long time to run (and in those cases you can generally split the test up into multiple tests, or comment out sections that are already working). The flow right now when developing mochitest for B2G is as follows: 1. Write test 2. Run mochitest in simulator 3. See test failures 4. Edit test 5. Go back to step 2. The problem here is that step 2 takes a very long time. It requires starting both gecko and the emulator. And it runs gecko under an ARM emulator. The turnaround time after having edited a test is several minutes. Which slows down test development quite dramatically when you have to wait several minutes for every missing parenthesis. This is actually a poor enough experience that unless we can figure out hacks to work around this, it probably makes sense to give up on this and put Bobby on writing code rather than tests. Another problem with the current setup is that once you run into a bug, you are forced to debug through the emulator rather than simply attaching to a normal desktop gecko process and use the same tools that we have years of polish and experience. As far as I understand it writing marionette tests have all the same problems. And would meant that we couldn't reuse the tests for Fennec and Desktop once these APIs work there too. Additionally marionette tests require writing mixed javascript and python and generally requires doing more complex things than mochitests currently do, especially for platform developers who are mostly experienced in javascript as well as the APIs they are testing. I'm definitely not complaining about marionette. It's solving a much more complex set of problems than mochitest is. But I don't think we can or should push them as an alternative to mochitest. Also note that writing Desktop Firefox mochitests is not an alternative since the relevant APIs doesn't exist there. How much work is involved with getting Mochitest running for desktop B2G? This would dramatically improve the turnaround time and actually make writing mochitests a real alternative. As things stand right now I don't feel that I can ask developers to write tests which is of course a very serious problem. Especially great would be if we make it possible to have the Desktop Firefox test writing flow where reloading a test is possible and you don't have to restart the whole test suite. / Jonas _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
