On 8/1/2013 1:18 AM, Jonas Sicking wrote:
Turning off these tests is not an option given how large percentage of
the B2G test body they represent. I've started sending some feelers to
see if I can find an owner.
While hiding the tests is not the same as turning them off, I agree that
to a developer the outcome is the same and we need a better way forward
here. I think we have other solutions available to us; it's a longer
discussion and so I'll start another thread on dev.platform about that.
Given the fact that the failing code here appears to be the marionette
code itself (unless I'm misunderstanding the bug), I had assumed that
this fell on the a-teams responsibility. Not sure if that's incorrect
or not?
This indeed does seem to be something at a low level where the b2g
system is interacting with marionette in an unusual way. As jgriffin
noted elsewhere, this doesn't happen on desktop builds, and so that
means it's not entirely marionette based. That said, he and dhylands
seem to have a handle on the issue in bug 898074 (at least a wallpaper
patch that removes the burning issue). We can take the wallpaper patch
that jgriffin just put up and this will enable us to avoid hiding the
tests, so the sheriffs can stand down.
All that said, this thread is much bigger than one automation bug.
There are several core issues here:
* Continual investigation into intermittent failures as they come up. If
we don't continue prioritizing these fixes, things escalate into
disasters very quickly. The most important one right now is the one
preventing the Gaia Unit tests from being un-hidden across all trees:
bug 898512
* Turning on more testing using existing frameworks - the Gaia Unit
tests are nearly ready to go, the Gaia UI tests are next in line (bug
897569), and the Gaia "integration" (Lal's JS framework) tests are after
that. This will help ensure we capture errors as they surface more
quickly, but catching errors doesn't do us any good if we don't
prioritize fixing them.
* Identifying where we are lacking test coverage and testability and
adding tests and testability into the product in those areas.
* Having QA step in and do manual and automated tests of master on
devices will continue to help (and it already is). QA is adding bugs
they find to the B2G-on-central tracking bug:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=884399&hide_resolved=1
Pushing efforts across all of these areas will be how we get
gaia-master/mozilla-central back to a better state.
Thanks,
Clint
On Wed, Jul 31, 2013 at 8:02 AM, Ryan VanderMeulen <[email protected]>
wrote:
On 7/25/2013 11:10 AM, Fabrice Desre wrote:
Tests are not in great shape either, and sheriffs are considering to
hide the B2G Marionnette tests later today due to the ongoing high
failure rate
(https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure%2C%20&keywords_type=allwords&short_desc=socket.timeout&resolution=---&query_format=advanced&short_desc_type=allwordssubstr&list_id=7335976)
FYI, I will be hiding the Marionette WebAPI test suite on m-c (and feeder
branches) on Friday since apparently nobody on the B2G team is going to own
getting them to satisfy our job visibility policy [1].
If someone wants to take ownership, bug 898074 is tracking the main source
of failures.
-Ryan
[1] https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy
_______________________________________________
dev-b2g mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-b2g
_______________________________________________
dev-b2g mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-b2g
_______________________________________________
dev-b2g mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-b2g
