I've also seen TV failures caused by flaws in the test harness which weren't visible before. See bug 1410165  for example, it was caused by an observer being unregistered after the first iteration of a test had finished, causing the following iterations to fail.
Gabriele  Permaorange test-verify dom/plugins/test/mochitest/test_crash_notify.xul | expected-crash-dump-missing - This test did not leave any crash dumps behind, but we were expecting some! https://bugzilla.mozilla.org/show_bug.cgi?id=1410165 On 06/03/18 20:05, Kyle Machulis wrote: > In my experience, many tests were written pre-test-verify, and don't clean > up correctly to deal with multiple runs in the same process. They work fine > when running as a single session, but blow up in TV. Having skip-if(verify) > means that we can at least mark those tests as known broken on TV without > taking them out of the test suites completely, especially if we find them > in places that we don't know how to fix them and need to file to someone > else who may not currently have time to fix. > > On Tue, Mar 6, 2018 at 10:49 AM, Emilio Cobos Álvarez <emi...@crisal.io> > wrote: > >> On 03/06/2018 06:04 PM, Geoffrey Brown wrote: >>> It is now possible to skip tests in test-verify. Simplify annotate the >>> manifest for your test: >>> >>> [test] >>> skip-if = verify >>> >>> or, for reftests: >>> >>> skip-if(verify) ... >>> >>> and the test-verify (TV) test task will not try to verify the annotated >>> test. >>> >>> Please don't abuse this feature! Most TV failures indicate a weakness in >>> the test. >> >> Could you point to an example of a "good" use of this feature? Is it >> just to avoid TV failing too intermittently? Is TV unable to run some >> tests? When does TV provide no value? >> >> Thanks! >> >> -- Emilio >> _______________________________________________ >> dev-platform mailing list >> firstname.lastname@example.org >> https://lists.mozilla.org/listinfo/dev-platform >> > _______________________________________________ > dev-platform mailing list > email@example.com > https://lists.mozilla.org/listinfo/dev-platform >
Description: OpenPGP digital signature
_______________________________________________ dev-platform mailing list firstname.lastname@example.org https://lists.mozilla.org/listinfo/dev-platform