On Wed, Dec 11, 2013 at 10:28 AM, Olivier Tilloy <[email protected]> wrote: > Is it common for the webbrowser-app tests to fail, be re-run, and then pass? > Does that happen weekly, daily, more often? Has it always been the case, or > did it start happening recently? > When it does happen, does the second run always pass, or how many re-runs > are necessary to reach the green state? Not *super* common, but it does happen from time to time. Last week, we re-ran a couple of webbrowser tests. All tests have become considerably more stable since the mtp fixes went in. Here are a couple with failures that we reran and got better results the second time last week: http://q-jenkins.ubuntu-ci:8080/job/trusty-touch-maguro-smoke-webbrowser-app-autopilot/47/ http://q-jenkins.ubuntu-ci:8080/job/trusty-touch-mako-smoke-webbrowser-app-autopilot/57/
>> For instance, here is on image 57 webbrowser with a lot of failing tests: >> http://ci.ubuntu.com/smokeng/trusty/touch/mako/57:20131210.1:20131203/5375/webbrowser-app-autopilot/. >> Would be interesting to know the root case, it seems something is screwed >> at some point, and then all tests are failing, can you have a look at that >> case and try to be more robust? (or pass to the CI team if it's an infra >> issue). > > > This specific run looks like an infrastructure issue to me, after a given > test all the subsequent tests fail when starting the app (it might be that > the app actually starts, but autopilot fails to find it). There are no crash > files in the artifacts, so it doesn’t look like an issue in the app itself. +Thomi - is there any way to tell the difference between the app not being started and autopilot not finding it? Unfortunately I don't have much to go on here either, but the maguro test of the same image passed fine and several builds around it passed, so it seems likely that mako would have passed if we reran it. Do the webbrowser tests depend on external websites or does it just connect locally on the device? -- Mailing list: https://launchpad.net/~ubuntu-phone Post to : [email protected] Unsubscribe : https://launchpad.net/~ubuntu-phone More help : https://help.launchpad.net/ListHelp
