On Wed, May 6, 2020 at 5:27 PM Joel Sherrill <j...@rtems.org> wrote: > > > > On Wed, May 6, 2020 at 6:12 AM Chris Johns <chr...@rtems.org> wrote: >> >> >> > On 6 May 2020, at 8:15 pm, Sebastian Huber >> > <sebastian.hu...@embedded-brains.de> wrote: >> > >> > On 06/05/2020 12:00, Chris Johns wrote: >> > >> >>> On 6/5/20 7:35 pm, Sebastian Huber wrote: >> >>>> On 06/05/2020 10:41, chr...@rtems.org wrote: >> >>> >> >>>> From: Chris Johns<chr...@rtems.org> >> >>>> >> >>>> Updates #2962 >> >>>> --- >> >>>> bsps/powerpc/psim/config/psim-testsuite.tcfg | 22 ++++++++++++++++++++ >> >>>> 1 file changed, 22 insertions(+) >> >>>> create mode 100644 bsps/powerpc/psim/config/psim-testsuite.tcfg >> >>>> >> >>>> diff --git a/bsps/powerpc/psim/config/psim-testsuite.tcfg >> >>>> b/bsps/powerpc/psim/config/psim-testsuite.tcfg >> >>>> new file mode 100644 >> >>>> index 0000000000..b0d2a05086 >> >>>> --- /dev/null >> >>>> +++ b/bsps/powerpc/psim/config/psim-testsuite.tcfg >> >>>> @@ -0,0 +1,22 @@ >> >>>> +# >> >>>> +# PSIM RTEMS Test Database. >> >>>> +# >> >>>> +# Format is one line per test that is_NOT_ built. >> >>>> +# >> >>>> + >> >>>> +expected-fail: fsimfsgeneric01 >> >>>> +expected-fail: block11 >> >>>> +expected-fail: rbheap01 >> >>>> +expected-fail: termios01 >> >>>> +expected-fail: ttest01 >> >>>> +expected-fail: psx12 >> >>>> +expected-fail: psxchroot01 >> >>>> +expected-fail: psxfenv01 >> >>>> +expected-fail: psximfs02 >> >>>> +expected-fail: psxpipe01 >> >>>> +expected-fail: spextensions01 >> >>>> +expected-fail: spfatal31 >> >>>> +expected-fail: spfifo02 >> >>>> +expected-fail: spmountmgr01 >> >>>> +expected-fail: spprivenv01 >> >>>> +expected-fail: spstdthreads01 >> >>> >> >>> I don't think these tests are expected to fail. If they fail, then there >> >>> is a bug somewhere. >> >> >> >> Yes we hope no tests fail but they can and do. Excluding tests because >> >> they fail would be incorrect. In the 5.1 release these bugs are present >> >> so we expect, or maybe it should say, we know the test will fail. With >> >> this change any thing that appears in the failure column is "unexpected" >> >> and that means the user build of the release does not match the state we >> >> "expect" and it is worth investigation by the user. >> >> >> >> Without these tests being tagged this way the user would have no idea >> >> where the stand after a build and test run and that would mean we would >> >> have to make sure a release has no failures. I consider that as not >> >> practical or realistic. >> > Maybe we need another state, e.g. something-is-broken-please-fix-it. >> >> I do not think so, it is implicit in the failure or the test is broken. The >> only change is to add unexpected-pass, that will be on master after the 5 >> branch. > > > This would be a good ticket to file as a "small tasks" one. Explain how to > find all of the expected failures and investigate if one can be fixed or > explained. >
That's perhaps the root of my complaint. By combining all "expected failures" under one label, we need to go back and fix up these when we fix a bug that makes it an "unexpected pass" I guess I understand this idea of "unexpected pass" now. I guess it doesn't really matter if we split the expected failures, but I think there is a difference that may be worth conveying to users. > For example, fenv support is missing for many architectures so those could > have a comment above them. But the jmr3904 has a different memory map from > most BSPs and I know one of the fatal error tests has the simulator catch the > invalid memory access before the RTEMS exception handler. That is really an > expected fail but deserves (and has I think) a comment above it in the tcfg > file. > > --joel > >> >> Chris >> >> Chris >> _______________________________________________ >> devel mailing list >> devel@rtems.org >> http://lists.rtems.org/mailman/listinfo/devel > > _______________________________________________ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel _______________________________________________ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
