A dedicated "citest" program is a good idea! The tests would need to be designed so that when a test passes, certain variables would have certain values, which can be tested to produce the PASS/FAIL result.
It will be tricky to design these tests, but it will be extremely valuable for verifying the OS across many CPU architectures. Having all tests PASS as a prerequisite for releases will give us a high confidence in NuttX. I think "citest" should not print any output, except in case of failed test, and a summary of number of PASSED, FAILED, and SKIPPED tests, which is printed at the end. But there should be a command line arg "--verbose" that causes it to report each PASS, FAIL, and SKIP as it happens. A test is SKIPPED when the architecture does not support the feature being tested, or when the OS is Kconfigured and built without support for a feature. The test will not run, but will report SKIPPED. Nathan On Thu, Aug 8, 2024 at 7:15 AM Tomek CEDRO <to...@cedro.info> wrote: > On Thu, Aug 8, 2024, 06:02 Gregory Nutt <spudan...@gmail.com> wrote: > > > > > On 8/7/2024 9:10 PM, yfliu2008 wrote: > > > ... if we had "ostest" based smoke testing coverage in our CI, those > > fake k230 issues can be prevented earlier. > > > > ostest would probably need some work to be useful in this case: ostest > > would need some kind of standard reporting in each test so that we would > > know when a test case passed or failed. > > > > Right now, many tests do not report any test results. That is because > > they run relatively complex, multi-threaded scenarios that are not easy > > to assess. You have to understand what the test is doing in order to > > interpret the test output. Such tests would need to report failures in > > a way that can be captured as a simple PASS or FAIL. > > > > how about new application citest dedicated just for that purpose ? > > it can reuse existing code parts from other places and by default just > print out something easy to parse i.e: > > [TESTX.Y.Z] [OK] Brief test description. > [TESTX.Y.Z][FA] Brief test description (error code). > > X.Y.Z could be code tree based test numbers i.e: > > X=1 for kernel > > Y=1 for IPC > > Z=1 as test number that does something > > configuration could include list of excluded tests and verbosity for > instance when test fails do we want to see the output why (or just include > error code in test brief). > > ? :-) > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > >
