Re: How to Classify Intermittent Test Failures

2021-02-02 Thread Chris Johns
On 3/2/21 3:13 am, Gedare Bloom wrote: > On Tue, Feb 2, 2021 at 7:40 AM Joel Sherrill > wrote: > On Mon, Feb 1, 2021 at 6:50 PM Chris Johns > wrote: > On 2/2/21 9:12 am, Joel Sherrill wrote: > >  On Mon, Feb 1, 2021 at 3:50

Re: How to Classify Intermittent Test Failures

2021-02-02 Thread Gedare Bloom
On Tue, Feb 2, 2021 at 7:40 AM Joel Sherrill wrote: > > > On Mon, Feb 1, 2021 at 6:50 PM Chris Johns wrote: > >> >> >> On 2/2/21 9:12 am, Joel Sherrill wrote: >> > On Mon, Feb 1, 2021 at 3:50 PM Chris Johns > > > wrote: >> > On 2/2/21 3:42 am, Joel Sherrill wrote:

Re: How to Classify Intermittent Test Failures

2021-02-02 Thread Joel Sherrill
On Mon, Feb 1, 2021 at 6:50 PM Chris Johns wrote: > > > On 2/2/21 9:12 am, Joel Sherrill wrote: > > On Mon, Feb 1, 2021 at 3:50 PM Chris Johns > > wrote: > > On 2/2/21 3:42 am, Joel Sherrill wrote: > > > Hi > > > > > > On the aarch64 qemu testing, we

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Chris Johns
On 2/2/21 9:12 am, Joel Sherrill wrote: > On Mon, Feb 1, 2021 at 3:50 PM Chris Johns > wrote: > On 2/2/21 3:42 am, Joel Sherrill wrote: > > Hi > > > > On the aarch64 qemu testing, we are seeing some tests which seem to pass > most of > > the

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Joel Sherrill
On Mon, Feb 1, 2021 at 3:50 PM Chris Johns wrote: > On 2/2/21 3:42 am, Joel Sherrill wrote: > > Hi > > > > On the aarch64 qemu testing, we are seeing some tests which seem to pass > most of > > the time but fail intermittently. It appears to be based somewhat on > host load > > but there may be

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Chris Johns
On 2/2/21 3:42 am, Joel Sherrill wrote: > Hi > > On the aarch64 qemu testing, we are seeing some tests which seem to pass most > of > the time but fail intermittently. It appears to be based somewhat on host load > but there may be other factors.  > > There does not appear to be a good test

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Gedare Bloom
On Mon, Feb 1, 2021 at 11:02 AM Joel Sherrill wrote: > > > On Mon, Feb 1, 2021 at 11:56 AM Peter Dufault wrote: > >> Can the test be marked as a possible emulator failure? For example (I >> haven't looked at the report) use "?" (Huh?) and not "!" (Failed!) or "." >> )(Passed), that is, the

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Joel Sherrill
On Mon, Feb 1, 2021 at 11:56 AM Peter Dufault wrote: > Can the test be marked as a possible emulator failure? For example (I > haven't looked at the report) use "?" (Huh?) and not "!" (Failed!) or "." > )(Passed), that is, the test failed as an emulated system and the RTEMS > project suspects

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Peter Dufault
Can the test be marked as a possible emulator failure? For example (I haven't looked at the report) use "?" (Huh?) and not "!" (Failed!) or "." )(Passed), that is, the test failed as an emulated system and the RTEMS project suspects the emulation itself is causing this. You shouldn't be

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Joel Sherrill
On Mon, Feb 1, 2021, 11:45 AM Gedare Bloom wrote: > > > On Mon, Feb 1, 2021 at 10:31 AM Joel Sherrill wrote: > >> >> >> On Mon, Feb 1, 2021 at 11:14 AM Gedare Bloom wrote: >> >>> >>> >>> On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill wrote: >>> Hi On the aarch64 qemu testing, we

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Gedare Bloom
On Mon, Feb 1, 2021 at 10:31 AM Joel Sherrill wrote: > > > On Mon, Feb 1, 2021 at 11:14 AM Gedare Bloom wrote: > >> >> >> On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill wrote: >> >>> Hi >>> >>> On the aarch64 qemu testing, we are seeing some tests which seem to pass >>> most of the time but fail

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Joel Sherrill
On Mon, Feb 1, 2021 at 11:14 AM Gedare Bloom wrote: > > > On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill wrote: > >> Hi >> >> On the aarch64 qemu testing, we are seeing some tests which seem to pass >> most of the time but fail intermittently. It appears to be based somewhat >> on host load but

Re: How to Classify Intermittent Test Failures

2021-02-01 Thread Gedare Bloom
On Mon, Feb 1, 2021 at 9:42 AM Joel Sherrill wrote: > Hi > > On the aarch64 qemu testing, we are seeing some tests which seem to pass > most of the time but fail intermittently. It appears to be based somewhat > on host load but there may be other factors. > > There does not appear to be a good

How to Classify Intermittent Test Failures

2021-02-01 Thread Joel Sherrill
Hi On the aarch64 qemu testing, we are seeing some tests which seem to pass most of the time but fail intermittently. It appears to be based somewhat on host load but there may be other factors. There does not appear to be a good test results state for these. Marking them expected pass or fail