I agree, I think it would take a long time to have proper coverage of NuttX as a whole. Perhaps even impossible, since Unity is more of a unit testing framework and thus there are definitely test cases for NuttX that would be outside of its realm of "applicability". There is still a need for a variety of testing applications and bench-marking applications for those scenarios.
That makes sense, prerequisites would need to be considered. I think targeting the low-hanging fruit of unit-testable code would be easy in that case, since prerequisites are mostly encoded in the Kconfig definitions existing or not for features to be tested. Any complex behaviour (interrupts, multi-threading, etc.) might be out of the realm of what's testable by Unity in the first place. We'll always have OS test in that case, but maybe then it's better to do what Alan suggested and at least make its output more clear in the meantime. I suppose RTOS testing is a whole field in itself. On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <[email protected]> wrote: > I think that the answer depends on a tradeoff between effort and test > coverage. I could imagine that a proper job could be man-years of effort. > > My only concern with a re-design is to be certain that tests are > arranged so that the prerequisites of later test are tested first. If > tests are grouped by, for example, functional area, then failures can be > difficult to understand if there is a failure in an unverified test > prerequisite. > > Some of the "tests" are bogus and only exist to provide some minimum > confidence that we have the functionality to run the test. For > example: Can I start tasks? Can I pass parameters? Can I survive a > context switch. Some of the tests were intended only to support debug. > Some were created only to track down specific bugs. > > On 12/21/25 18:39, Matteo Golin wrote: > > LTS being the Long-Term-Stable release of NuttX? > > > > I'm starting to think that it would be best to create new suite of tests > > using Unity, using what's in OS Test as a basis and then expanding it > > further. Greg has pointed out that OS test is not intended to be a > complete > > test, so maybe a more solid testing framework base in a new form would be > > in order. At that point it would also be good to document and educate > users > > about other test options, since OS test is a current favourite for > showing > > PRs haven't broken anything. Maybe that's not the best choice anymore > with > > this information. > > > > I don't want to re-invent the wheel of what's already in `testsuite`, > but I > > think Unity might end up being lighter-weight in the long run? And it is > > also not a Xiaomi owned/maintained framework so it may receive better > > external support (i.e. more users rely on it?). I tried to compile the > > sim:citest configuration to run the testsuite example and see what it was > > like, but I was met with a long list of C++ compilation errors that I'd > > rather not deal with. There also seems to be zero docs about it or cmocka > > in our NuttX docs at the moment. > > > > What do you think? Start a Unity test suite, add some switches to choose > > which test cases get compiled into the binary, open for extensions by > users > > who know more about different subsystems? This way we can keep OS test > how > > it is for a quick, low-resource sanity check of "board bringup" while > > having a dedicated testing application for patches? Someone changing file > > system stuff can just compile to include file system test cases and run > > that suite only as proof that their PR works. The output is really easy > to > > glance at to determine pass/fail results. > > > > Anyways, here's my draft demonstrating how Unity works with the FPU test > > suite from OS test: https://github.com/apache/nuttx-apps/pull/3259 > > > > Matteo > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <[email protected]> wrote: > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <[email protected]> > wrote: > >>> (..) > >>> Starting with bits and pieces of something like the LTS might make more > >>> sense other than its licensing problems. Since it would never be > >>> included in a distributed binary, the licensing really should not > matter > >>> much in practice. > >> Unless its GPL and some company would like to have their own internal > >> bits and pieces.. luckily its MIT and the whole thing is open-source > >> anyways :-) > >> > >> Yes we had many discussions about the LTS and this could be great start! > >> :-) > >> > >> -- > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > >> >
