Re: skip-if(verify)
I've also seen TV failures caused by flaws in the test harness which weren't visible before. See bug 1410165 [1] for example, it was caused by an observer being unregistered after the first iteration of a test had finished, causing the following iterations to fail. Gabriele [1] Permaorange test-verify dom/plugins/test/mochitest/test_crash_notify.xul | expected-crash-dump-missing - This test did not leave any crash dumps behind, but we were expecting some! https://bugzilla.mozilla.org/show_bug.cgi?id=1410165 On 06/03/18 20:05, Kyle Machulis wrote: > In my experience, many tests were written pre-test-verify, and don't clean > up correctly to deal with multiple runs in the same process. They work fine > when running as a single session, but blow up in TV. Having skip-if(verify) > means that we can at least mark those tests as known broken on TV without > taking them out of the test suites completely, especially if we find them > in places that we don't know how to fix them and need to file to someone > else who may not currently have time to fix. > > On Tue, Mar 6, 2018 at 10:49 AM, Emilio Cobos Álvarez > wrote: > >> On 03/06/2018 06:04 PM, Geoffrey Brown wrote: >>> It is now possible to skip tests in test-verify. Simplify annotate the >>> manifest for your test: >>> >>> [test] >>> skip-if = verify >>> >>> or, for reftests: >>> >>> skip-if(verify) ... >>> >>> and the test-verify (TV) test task will not try to verify the annotated >>> test. >>> >>> Please don't abuse this feature! Most TV failures indicate a weakness in >>> the test. >> >> Could you point to an example of a "good" use of this feature? Is it >> just to avoid TV failing too intermittently? Is TV unable to run some >> tests? When does TV provide no value? >> >> Thanks! >> >> -- Emilio >> ___ >> dev-platform mailing list >> dev-platform@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-platform >> > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > signature.asc Description: OpenPGP digital signature ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: skip-if(verify)
On 03/06/2018 08:05 PM, Kyle Machulis wrote: > In my experience, many tests were written pre-test-verify, and don't clean > up correctly to deal with multiple runs in the same process. They work fine > when running as a single session, but blow up in TV. Having skip-if(verify) > means that we can at least mark those tests as known broken on TV without > taking them out of the test suites completely, especially if we find them > in places that we don't know how to fix them and need to file to someone > else who may not currently have time to fix. Ahá, that makes perfect sense, thanks a lot! -- Emilio > On Tue, Mar 6, 2018 at 10:49 AM, Emilio Cobos Álvarez > wrote: > >> On 03/06/2018 06:04 PM, Geoffrey Brown wrote: >>> It is now possible to skip tests in test-verify. Simplify annotate the >>> manifest for your test: >>> >>> [test] >>> skip-if = verify >>> >>> or, for reftests: >>> >>> skip-if(verify) ... >>> >>> and the test-verify (TV) test task will not try to verify the annotated >>> test. >>> >>> Please don't abuse this feature! Most TV failures indicate a weakness in >>> the test. >> >> Could you point to an example of a "good" use of this feature? Is it >> just to avoid TV failing too intermittently? Is TV unable to run some >> tests? When does TV provide no value? >> >> Thanks! >> >> -- Emilio >> ___ >> dev-platform mailing list >> dev-platform@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-platform >> > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: skip-if(verify)
In my experience, many tests were written pre-test-verify, and don't clean up correctly to deal with multiple runs in the same process. They work fine when running as a single session, but blow up in TV. Having skip-if(verify) means that we can at least mark those tests as known broken on TV without taking them out of the test suites completely, especially if we find them in places that we don't know how to fix them and need to file to someone else who may not currently have time to fix. On Tue, Mar 6, 2018 at 10:49 AM, Emilio Cobos Álvarez wrote: > On 03/06/2018 06:04 PM, Geoffrey Brown wrote: > > It is now possible to skip tests in test-verify. Simplify annotate the > > manifest for your test: > > > > [test] > > skip-if = verify > > > > or, for reftests: > > > > skip-if(verify) ... > > > > and the test-verify (TV) test task will not try to verify the annotated > > test. > > > > Please don't abuse this feature! Most TV failures indicate a weakness in > > the test. > > Could you point to an example of a "good" use of this feature? Is it > just to avoid TV failing too intermittently? Is TV unable to run some > tests? When does TV provide no value? > > Thanks! > > -- Emilio > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: skip-if(verify)
On 03/06/2018 06:04 PM, Geoffrey Brown wrote: > It is now possible to skip tests in test-verify. Simplify annotate the > manifest for your test: > > [test] > skip-if = verify > > or, for reftests: > > skip-if(verify) ... > > and the test-verify (TV) test task will not try to verify the annotated > test. > > Please don't abuse this feature! Most TV failures indicate a weakness in > the test. Could you point to an example of a "good" use of this feature? Is it just to avoid TV failing too intermittently? Is TV unable to run some tests? When does TV provide no value? Thanks! -- Emilio ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
skip-if(verify)
It is now possible to skip tests in test-verify. Simplify annotate the manifest for your test: [test] skip-if = verify or, for reftests: skip-if(verify) ... and the test-verify (TV) test task will not try to verify the annotated test. Please don't abuse this feature! Most TV failures indicate a weakness in the test. As always, you can read more about test-verify at https://developer.mozilla.org/en-US/docs/Mozilla/QA/Test_Verification. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform