On Wed, May 6, 2020 at 5:12 AM Chris Johns <chr...@rtems.org> wrote: > > > > On 6 May 2020, at 8:15 pm, Sebastian Huber > > <sebastian.hu...@embedded-brains.de> wrote: > > > > On 06/05/2020 12:00, Chris Johns wrote: > > > >>> On 6/5/20 7:35 pm, Sebastian Huber wrote: > >>>> On 06/05/2020 10:41, chr...@rtems.org wrote: > >>> > >>>> From: Chris Johns<chr...@rtems.org> > >>>> > >>>> Updates #2962 > >>>> --- > >>>> bsps/powerpc/psim/config/psim-testsuite.tcfg | 22 ++++++++++++++++++++ > >>>> 1 file changed, 22 insertions(+) > >>>> create mode 100644 bsps/powerpc/psim/config/psim-testsuite.tcfg > >>>> > >>>> diff --git a/bsps/powerpc/psim/config/psim-testsuite.tcfg > >>>> b/bsps/powerpc/psim/config/psim-testsuite.tcfg > >>>> new file mode 100644 > >>>> index 0000000000..b0d2a05086 > >>>> --- /dev/null > >>>> +++ b/bsps/powerpc/psim/config/psim-testsuite.tcfg > >>>> @@ -0,0 +1,22 @@ > >>>> +# > >>>> +# PSIM RTEMS Test Database. > >>>> +# > >>>> +# Format is one line per test that is_NOT_ built. > >>>> +# > >>>> + > >>>> +expected-fail: fsimfsgeneric01 > >>>> +expected-fail: block11 > >>>> +expected-fail: rbheap01 > >>>> +expected-fail: termios01 > >>>> +expected-fail: ttest01 > >>>> +expected-fail: psx12 > >>>> +expected-fail: psxchroot01 > >>>> +expected-fail: psxfenv01 > >>>> +expected-fail: psximfs02 > >>>> +expected-fail: psxpipe01 > >>>> +expected-fail: spextensions01 > >>>> +expected-fail: spfatal31 > >>>> +expected-fail: spfifo02 > >>>> +expected-fail: spmountmgr01 > >>>> +expected-fail: spprivenv01 > >>>> +expected-fail: spstdthreads01 > >>> > >>> I don't think these tests are expected to fail. If they fail, then there > >>> is a bug somewhere. > >> > >> Yes we hope no tests fail but they can and do. Excluding tests because > >> they fail would be incorrect. In the 5.1 release these bugs are present so > >> we expect, or maybe it should say, we know the test will fail. With this > >> change any thing that appears in the failure column is "unexpected" and > >> that means the user build of the release does not match the state we > >> "expect" and it is worth investigation by the user. > >> > >> Without these tests being tagged this way the user would have no idea > >> where the stand after a build and test run and that would mean we would > >> have to make sure a release has no failures. I consider that as not > >> practical or realistic. > > Maybe we need another state, e.g. something-is-broken-please-fix-it. > > I do not think so, it is implicit in the failure or the test is broken. The > only change is to add unexpected-pass, that will be on master after the 5 > branch. >
I disagree with this in principle, and it should be reverted after we branch 5. It's fine for now to get the release state sync'd, but we should find a long-term solution that distinguishes the cases: 1. we don't expect this test to pass on this bsp 2. we expect this test to pass, but know it doesn't currently They are two very different things, and I don't like conflating them into one "expected-fail" case > Chris > > Chris > _______________________________________________ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
