Small steps with measurable results may be the best approach here, to create some new solution next to existing ostest but not replacing it until ready? :-)
-- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <[email protected]> wrote: > > I agree, I think it would take a long time to have proper coverage of NuttX > as a whole. Perhaps even impossible, since Unity is more of a unit testing > framework and thus there are definitely test cases for NuttX that would be > outside of its realm of "applicability". There is still a need for a > variety of testing applications and bench-marking applications for those > scenarios. > > That makes sense, prerequisites would need to be considered. I think > targeting the low-hanging fruit of unit-testable code would be easy in that > case, since prerequisites are mostly encoded in the Kconfig definitions > existing or not for features to be tested. Any complex behaviour > (interrupts, multi-threading, etc.) might be out of the realm of what's > testable by Unity in the first place. We'll always have OS test in that > case, but maybe then it's better to do what Alan suggested and at least > make its output more clear in the meantime. I suppose RTOS testing is a > whole field in itself. > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <[email protected]> wrote: > > > I think that the answer depends on a tradeoff between effort and test > > coverage. I could imagine that a proper job could be man-years of effort. > > > > My only concern with a re-design is to be certain that tests are > > arranged so that the prerequisites of later test are tested first. If > > tests are grouped by, for example, functional area, then failures can be > > difficult to understand if there is a failure in an unverified test > > prerequisite. > > > > Some of the "tests" are bogus and only exist to provide some minimum > > confidence that we have the functionality to run the test. For > > example: Can I start tasks? Can I pass parameters? Can I survive a > > context switch. Some of the tests were intended only to support debug. > > Some were created only to track down specific bugs. > > > > On 12/21/25 18:39, Matteo Golin wrote: > > > LTS being the Long-Term-Stable release of NuttX? > > > > > > I'm starting to think that it would be best to create new suite of tests > > > using Unity, using what's in OS Test as a basis and then expanding it > > > further. Greg has pointed out that OS test is not intended to be a > > complete > > > test, so maybe a more solid testing framework base in a new form would be > > > in order. At that point it would also be good to document and educate > > users > > > about other test options, since OS test is a current favourite for > > showing > > > PRs haven't broken anything. Maybe that's not the best choice anymore > > with > > > this information. > > > > > > I don't want to re-invent the wheel of what's already in `testsuite`, > > but I > > > think Unity might end up being lighter-weight in the long run? And it is > > > also not a Xiaomi owned/maintained framework so it may receive better > > > external support (i.e. more users rely on it?). I tried to compile the > > > sim:citest configuration to run the testsuite example and see what it was > > > like, but I was met with a long list of C++ compilation errors that I'd > > > rather not deal with. There also seems to be zero docs about it or cmocka > > > in our NuttX docs at the moment. > > > > > > What do you think? Start a Unity test suite, add some switches to choose > > > which test cases get compiled into the binary, open for extensions by > > users > > > who know more about different subsystems? This way we can keep OS test > > how > > > it is for a quick, low-resource sanity check of "board bringup" while > > > having a dedicated testing application for patches? Someone changing file > > > system stuff can just compile to include file system test cases and run > > > that suite only as proof that their PR works. The output is really easy > > to > > > glance at to determine pass/fail results. > > > > > > Anyways, here's my draft demonstrating how Unity works with the FPU test > > > suite from OS test: https://github.com/apache/nuttx-apps/pull/3259 > > > > > > Matteo > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <[email protected]> wrote: > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <[email protected]> > > wrote: > > >>> (..) > > >>> Starting with bits and pieces of something like the LTS might make more > > >>> sense other than its licensing problems. Since it would never be > > >>> included in a distributed binary, the licensing really should not > > matter > > >>> much in practice. > > >> Unless its GPL and some company would like to have their own internal > > >> bits and pieces.. luckily its MIT and the whole thing is open-source > > >> anyways :-) > > >> > > >> Yes we had many discussions about the LTS and this could be great start! > > >> :-) > > >> > > >> -- > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > >> > >
