Hi Jonas,
I don't think it would take much work to add support for running b2g
mochitests on the b2g desktop build, it just hasn't been a priority
since we're not running tests in that environment for TBPL, and there
hasn't been much interest from developers, at least that I've heard.
However, if it would be a big help, I'd be happy to tackle that.
Additionally marionette
tests require writing mixed javascript and python and generally
requires doing more complex things than mochitests currently do,
especially for platform developers who are mostly experienced in
javascript as well as the APIs they are testing.
That isn't accurate; all of the WebAPI tests are JavaScript-only (e.g.,
http://mxr.mozilla.org/mozilla-central/source/dom/sms/tests/marionette/test_incoming.js).
However, your point is well-taken, that Marionette was never intended as
a mochitest replacement, but rather as a harness that allows us to do
things that mochitest can't. For things that mochitest can do, I agree
we should continue to use it.
I'll need to look at it before I can give an ETA, but you can follow
https://bugzilla.mozilla.org/show_bug.cgi?id=826111 for progress.
Regards,
Jonathan
On 1/3/2013 7:29 PM, Jonas Sicking wrote:
Hi All,
As was discussed in the convergence meeting wednesday, we are severely
lacking tests.
So I asked Bobby Holley to write some tests for the install/update API
that we have had quite a few problems with lately. Both regarding
regressions, and regarding various edge and error cases not working.
Unfortunately he very quickly ran into the problem that basically the
current workflow is so hard to use that it's almost essentially
broken. Which is likely why we have essentially no mochitests.
The flow when developing a mochitest on desktop is something like the following:
1. Write test
2. Run Firefox such that it opens the single mochitest test that is
being developed
3. See test failures
4. Edit test
5. Reload test in Firefox (which is still open since step 2)
6. Go back to step 3.
This means that the turnaround time after having edited a test is
basically 0. Reloading is instant except in the rare cases when the
test takes a long time to run (and in those cases you can generally
split the test up into multiple tests, or comment out sections that
are already working).
The flow right now when developing mochitest for B2G is as follows:
1. Write test
2. Run mochitest in simulator
3. See test failures
4. Edit test
5. Go back to step 2.
The problem here is that step 2 takes a very long time. It requires
starting both gecko and the emulator. And it runs gecko under an ARM
emulator. The turnaround time after having edited a test is several
minutes. Which slows down test development quite dramatically when you
have to wait several minutes for every missing parenthesis.
This is actually a poor enough experience that unless we can figure
out hacks to work around this, it probably makes sense to give up on
this and put Bobby on writing code rather than tests.
Another problem with the current setup is that once you run into a
bug, you are forced to debug through the emulator rather than simply
attaching to a normal desktop gecko process and use the same tools
that we have years of polish and experience.
As far as I understand it writing marionette tests have all the same
problems. And would meant that we couldn't reuse the tests for Fennec
and Desktop once these APIs work there too. Additionally marionette
tests require writing mixed javascript and python and generally
requires doing more complex things than mochitests currently do,
especially for platform developers who are mostly experienced in
javascript as well as the APIs they are testing. I'm definitely not
complaining about marionette. It's solving a much more complex set of
problems than mochitest is. But I don't think we can or should push
them as an alternative to mochitest.
Also note that writing Desktop Firefox mochitests is not an
alternative since the relevant APIs doesn't exist there.
How much work is involved with getting Mochitest running for desktop
B2G? This would dramatically improve the turnaround time and actually
make writing mochitests a real alternative. As things stand right now
I don't feel that I can ask developers to write tests which is of
course a very serious problem.
Especially great would be if we make it possible to have the Desktop
Firefox test writing flow where reloading a test is possible and you
don't have to restart the whole test suite.
/ Jonas
_______________________________________________
dev-b2g mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-b2g
