Re: skip-if(verify)

[Gabriele Svelto]

I've also seen TV failures caused by flaws in the test harness which
weren't visible before. See bug 1410165 [1] for example, it was caused
by an observer being unregistered after the first iteration of a test
had finished, causing the following iterations to fail.

 Gabriele

[1] Permaorange test-verify
dom/plugins/test/mochitest/test_crash_notify.xul |
expected-crash-dump-missing - This test did not leave any crash dumps
behind, but we were expecting some!
https://bugzilla.mozilla.org/show_bug.cgi?id=1410165

On 06/03/18 20:05, Kyle Machulis wrote:
> In my experience, many tests were written pre-test-verify, and don't clean
> up correctly to deal with multiple runs in the same process. They work fine
> when running as a single session, but blow up in TV. Having skip-if(verify)
> means that we can at least mark those tests as known broken on TV without
> taking them out of the test suites completely, especially if we find them
> in places that we don't know how to fix them and need to file to someone
> else who may not currently have time to fix.
> 
> On Tue, Mar 6, 2018 at 10:49 AM, Emilio Cobos Álvarez 
> wrote:
> 
>> On 03/06/2018 06:04 PM, Geoffrey Brown wrote:
>>> It is now possible to skip tests in test-verify. Simplify annotate the
>>> manifest for your test:
>>>
>>> [test]
>>> skip-if = verify
>>>
>>> or, for reftests:
>>>
>>> skip-if(verify) ...
>>>
>>> and the test-verify (TV) test task will not try to verify the annotated
>>> test.
>>>
>>> Please don't abuse this feature! Most TV failures indicate a weakness in
>>> the test.
>>
>> Could you point to an example of a "good" use of this feature? Is it
>> just to avoid TV failing too intermittently? Is TV unable to run some
>> tests? When does TV provide no value?
>>
>> Thanks!
>>
>>  -- Emilio
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: skip-if(verify)

[Emilio Cobos Álvarez]

On 03/06/2018 08:05 PM, Kyle Machulis wrote:
> In my experience, many tests were written pre-test-verify, and don't clean
> up correctly to deal with multiple runs in the same process. They work fine
> when running as a single session, but blow up in TV. Having skip-if(verify)
> means that we can at least mark those tests as known broken on TV without
> taking them out of the test suites completely, especially if we find them
> in places that we don't know how to fix them and need to file to someone
> else who may not currently have time to fix.

Ahá, that makes perfect sense, thanks a lot!

 -- Emilio

> On Tue, Mar 6, 2018 at 10:49 AM, Emilio Cobos Álvarez 
> wrote:
> 
>> On 03/06/2018 06:04 PM, Geoffrey Brown wrote:
>>> It is now possible to skip tests in test-verify. Simplify annotate the
>>> manifest for your test:
>>>
>>> [test]
>>> skip-if = verify
>>>
>>> or, for reftests:
>>>
>>> skip-if(verify) ...
>>>
>>> and the test-verify (TV) test task will not try to verify the annotated
>>> test.
>>>
>>> Please don't abuse this feature! Most TV failures indicate a weakness in
>>> the test.
>>
>> Could you point to an example of a "good" use of this feature? Is it
>> just to avoid TV failing too intermittently? Is TV unable to run some
>> tests? When does TV provide no value?
>>
>> Thanks!
>>
>>  -- Emilio
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: skip-if(verify)

[Kyle Machulis]

In my experience, many tests were written pre-test-verify, and don't clean
up correctly to deal with multiple runs in the same process. They work fine
when running as a single session, but blow up in TV. Having skip-if(verify)
means that we can at least mark those tests as known broken on TV without
taking them out of the test suites completely, especially if we find them
in places that we don't know how to fix them and need to file to someone
else who may not currently have time to fix.

On Tue, Mar 6, 2018 at 10:49 AM, Emilio Cobos Álvarez 
wrote:

> On 03/06/2018 06:04 PM, Geoffrey Brown wrote:
> > It is now possible to skip tests in test-verify. Simplify annotate the
> > manifest for your test:
> >
> > [test]
> > skip-if = verify
> >
> > or, for reftests:
> >
> > skip-if(verify) ...
> >
> > and the test-verify (TV) test task will not try to verify the annotated
> > test.
> >
> > Please don't abuse this feature! Most TV failures indicate a weakness in
> > the test.
>
> Could you point to an example of a "good" use of this feature? Is it
> just to avoid TV failing too intermittently? Is TV unable to run some
> tests? When does TV provide no value?
>
> Thanks!
>
>  -- Emilio
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: skip-if(verify)

[Emilio Cobos Álvarez]

On 03/06/2018 06:04 PM, Geoffrey Brown wrote:
> It is now possible to skip tests in test-verify. Simplify annotate the
> manifest for your test:
> 
> [test]
> skip-if = verify
> 
> or, for reftests:
> 
> skip-if(verify) ...
> 
> and the test-verify (TV) test task will not try to verify the annotated
> test.
> 
> Please don't abuse this feature! Most TV failures indicate a weakness in
> the test.

Could you point to an example of a "good" use of this feature? Is it
just to avoid TV failing too intermittently? Is TV unable to run some
tests? When does TV provide no value?

Thanks!

 -- Emilio
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform