Would separating intermittent test jobs into their own separate job help here? That way the retry logic can be removed for the more stable tests.
On 17 December 2015 at 14:00, Michael Henretty <[email protected]> wrote: > Resurrecting this old thread since intermittents are still a problem, and > now that we are back from Mozlando I want to have a clear path forward here. > > As a quick refresher, the retry logic inside the Gij test runner makes it > very hard to know which tests are the worst intermittents. In bug 1222215, > we have been trying to mark and fix all the intermittents we find so that > we can disable this retry logic. But this has proven very time consuming, > and in the meantime new intermittents can be introduced and not noticed due > to bug 1222215. > > Since these intermittents slow down development for everybody, are a > general annoyance to sheriffs and devs alike, and since they provide us > with little trust in protecting our features, here is the path forward we > are going to take: > > - Disable all current intermittents that block bug 1222215. > - Land bug 1222215. > - Focus efforts on re-enabling the intermittent tests. > > This way, we will at least be protected against introducing new racy tests > while we fix the old ones. I have spoken to Julie McCracken, and she agrees > with this way forward. We will be ni? module owners on these tests to get > help re-enabling them. Also, if you want to help out with this effort > please reach out to me or Julie. Let's take this opportunity to get our > current tests nice and green while we figure out the direction forward for > Firefox OS. > > Thanks! > Michael > > > > > On Wed, Nov 18, 2015 at 7:02 PM, Jonas Sicking <[email protected]> wrote: > >> Great to hear that socket disconnects aren't a big problem any more. >> That is really really awesome!! >> >> As far as synchronous vs. async goes, I'd say that optimizing for >> making it easy to write tests is generally much more important than >> optimizing for getting tests to run fast. >> >> This is especially true if slowness happens on desktop computers >> running the test harness, rather than on the phone hardware that we >> are testing. Desktop computers are fairly cheap to scale up in >> comparison to engineering manpower. >> >> And engineers (me included) tend to dislike test writing and test >> debugging enough that anything we can do to make that more pleasant >> has a big payoff in the quality and quantity of tests that we have. >> >> / Jonas >> >> >> On Wed, Nov 18, 2015 at 2:29 AM, Michael Henretty <[email protected]> >> wrote: >> > >> > On Wed, Nov 18, 2015 at 1:47 AM, Aus Lacroix <[email protected]> wrote: >> >> >> >> Jonas, that particular error is on the decline. Many went away when we >> >> rolled out a series of a fixes to run the tests on devices. The error >> itself >> >> was a symptom of a different issue. I would imagine that the ones that >> we >> >> still see occurring are, likely, also not directly related to >> sockit-to-me. >> > >> > >> > FWIW, in working on bug 1222215 I haven't seen the dreaded "Cannot call >> send >> > of undefined" error once. I didn't want to jinx it, but I think we can >> > safely say this is no longer an issue for us (knock on wood). >> > >> >> >> >> >> >> Even though this is the case, we recognize that synchronous tcp socket >> >> usage isn't ideal (we didn't think it was in the first place, >> necessarily, >> >> it was just the best way to make the tests easy to write). >> >> >> >> FFWD to now, we're adding a promise based tcp driver for marionette >> which >> >> will enable new tests to be written using promises. Marionette calls >> would >> >> always return a promise which you could .then() to do something else. >> It's a >> >> much nicer, and standardized pattern. >> > >> > >> > Putting in my two cents, I love the fact that sockit-to-me is >> synchronous. >> > It allows me to read, write, and understand tests very quickly. I >> realize >> > there are performance problems with a busy wait, but I don't think this >> is a >> > problem during marionette tests since they aren't user facing. What >> other >> > advantages does switching to a Promise based driver give us? Do we >> think the >> > tests will be less intermittent? >> > >> > > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos > > -- Zambrano Gasparnian, Armen Automation & Tools Engineer http://armenzg.blogspot.ca
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
