I think so. Best to leave OS test alone and slowly replace it with a new tool that has improvements. The only reason I can see to modify OS test now would just be to have clearer output about what passed/failed.
I'll come up with an improved proof of concept for a new tool to share in the next few days. Matteo On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <[email protected]> wrote: > Small steps with measurable results may be the best approach here, to > create some new solution next to existing ostest but not replacing it > until ready? :-) > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <[email protected]> > wrote: > > > > I agree, I think it would take a long time to have proper coverage of > NuttX > > as a whole. Perhaps even impossible, since Unity is more of a unit > testing > > framework and thus there are definitely test cases for NuttX that would > be > > outside of its realm of "applicability". There is still a need for a > > variety of testing applications and bench-marking applications for those > > scenarios. > > > > That makes sense, prerequisites would need to be considered. I think > > targeting the low-hanging fruit of unit-testable code would be easy in > that > > case, since prerequisites are mostly encoded in the Kconfig definitions > > existing or not for features to be tested. Any complex behaviour > > (interrupts, multi-threading, etc.) might be out of the realm of what's > > testable by Unity in the first place. We'll always have OS test in that > > case, but maybe then it's better to do what Alan suggested and at least > > make its output more clear in the meantime. I suppose RTOS testing is a > > whole field in itself. > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <[email protected]> > wrote: > > > > > I think that the answer depends on a tradeoff between effort and test > > > coverage. I could imagine that a proper job could be man-years of > effort. > > > > > > My only concern with a re-design is to be certain that tests are > > > arranged so that the prerequisites of later test are tested first. If > > > tests are grouped by, for example, functional area, then failures can > be > > > difficult to understand if there is a failure in an unverified test > > > prerequisite. > > > > > > Some of the "tests" are bogus and only exist to provide some minimum > > > confidence that we have the functionality to run the test. For > > > example: Can I start tasks? Can I pass parameters? Can I survive a > > > context switch. Some of the tests were intended only to support debug. > > > Some were created only to track down specific bugs. > > > > > > On 12/21/25 18:39, Matteo Golin wrote: > > > > LTS being the Long-Term-Stable release of NuttX? > > > > > > > > I'm starting to think that it would be best to create new suite of > tests > > > > using Unity, using what's in OS Test as a basis and then expanding it > > > > further. Greg has pointed out that OS test is not intended to be a > > > complete > > > > test, so maybe a more solid testing framework base in a new form > would be > > > > in order. At that point it would also be good to document and educate > > > users > > > > about other test options, since OS test is a current favourite for > > > showing > > > > PRs haven't broken anything. Maybe that's not the best choice anymore > > > with > > > > this information. > > > > > > > > I don't want to re-invent the wheel of what's already in `testsuite`, > > > but I > > > > think Unity might end up being lighter-weight in the long run? And > it is > > > > also not a Xiaomi owned/maintained framework so it may receive better > > > > external support (i.e. more users rely on it?). I tried to compile > the > > > > sim:citest configuration to run the testsuite example and see what > it was > > > > like, but I was met with a long list of C++ compilation errors that > I'd > > > > rather not deal with. There also seems to be zero docs about it or > cmocka > > > > in our NuttX docs at the moment. > > > > > > > > What do you think? Start a Unity test suite, add some switches to > choose > > > > which test cases get compiled into the binary, open for extensions by > > > users > > > > who know more about different subsystems? This way we can keep OS > test > > > how > > > > it is for a quick, low-resource sanity check of "board bringup" while > > > > having a dedicated testing application for patches? Someone changing > file > > > > system stuff can just compile to include file system test cases and > run > > > > that suite only as proof that their PR works. The output is really > easy > > > to > > > > glance at to determine pass/fail results. > > > > > > > > Anyways, here's my draft demonstrating how Unity works with the FPU > test > > > > suite from OS test: https://github.com/apache/nuttx-apps/pull/3259 > > > > > > > > Matteo > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <[email protected]> > wrote: > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <[email protected]> > > > wrote: > > > >>> (..) > > > >>> Starting with bits and pieces of something like the LTS might make > more > > > >>> sense other than its licensing problems. Since it would never be > > > >>> included in a distributed binary, the licensing really should not > > > matter > > > >>> much in practice. > > > >> Unless its GPL and some company would like to have their own > internal > > > >> bits and pieces.. luckily its MIT and the whole thing is open-source > > > >> anyways :-) > > > >> > > > >> Yes we had many discussions about the LTS and this could be great > start! > > > >> :-) > > > >> > > > >> -- > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > >> > > > >
