The discussion about small systems doesn't make much sense without data comparing resources of the same test cases for cmock and unity.
But let's assume that Unity is better. Should we use two frameworks to implement unit tests for OS? Or choose one framework? If we choose only Unity, who will migrate the current cmocka tests to Unity? Remember that cmoka tests are used in CI, so all CI tools must be also migrated. Or should we use two different frameworks to test different things and make even more mess in apps/testing as it is now? No one is forcing anyone to do anything. This is an open-source project, everyone should do what they feel is right. But whether the change should be merged is a different question. In this case, it's a very important question, because unit testing isn't just about test cases. It's about the entire infrastructure that runs, captures and handles test results later. Writing the test cases is the least of the problems. The infrastructure for cmoka testing is already in place, the infrastructure for Unity testing isn't. pon., 22 gru 2025 o 14:16 Alan C. Assis <[email protected]> napisał(a): > Mateusz, > but some chips like AVR DA have big flash, but not so much RAM. > > It is better to have some possible way to test on a small system, using > cmocka we are removing this option from the table. > > I liked the test Matteo did with Unity and I think we should let him > continue. Forcing him to migrate to cmocka just because there's a test > suite from Xiaomi doesn't seem prudent to me. > > This is the advantage of an open mind open-source project: "the possibility > to test new ideas", if someone had said to Michal Lenc he couldn't create > NXBoot because where is already MCUBoot integrated on NuttX, we didn't have > the option to use NXBoot today. ;-) > > BR, > > Alan > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <[email protected]> wrote: > > > Unit testing by itself isn't suitable for small systems. Ostest is > > something > > different (system level testing), so running it on small systems makes > > sense. > > Running unit tests on small systems is pointless because you won't fit in > > FLASH > > either way. You can run the same on large targets (preferably on host: > sim, > > qemu) > > and achieve the same result. > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <[email protected]> napisał(a): > > > > > Is it too complicated to rewrite the cmocka tests in Unity? > > > > > > If cmoka is not feasible for small microcontrollers, like AVRs and > > others, > > > our test coverage will be faulty. > > > > > > BR, > > > > > > Alan > > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <[email protected]> > wrote: > > > > > > > I'm a fan of Unity and have been using it in my projects for years, > > but I > > > > think > > > > we should stick with one testing framework. Currently, Cmocka is used > > in > > > > nuttx-apps, and we should stick with it. The only question is how > well > > > > Cmocka > > > > is suitable for running on small systems? Will ostest still be able > to > > > run > > > > on small MCUs? > > > > With Unity this would be quite likely, so maybe an exception for > Unity > > > and > > > > ostest would > > > > be acceptable, but using it for other test cases will only cause > chaos. > > > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> napisał(a): > > > > > > > > > Hi Matteo, > > > > > > > > > > Good idea ! > > > > > > > > > > But it is best to stick with a single testing framework. Currently, > > > many > > > > > test cases are using cmocka, so I suggest reusing cmocka for > > > consistency. > > > > > > > > > > ostest is a minimal basic test suite, typically used for CT > > (continuous > > > > > testing) of each patch. > > > > > If we move ostest to cmocka, it would make ostest dependent on a > > > specific > > > > > framework before it can be run. > > > > > And It may increase the code size, which is not very friendly for > > > > > configurations with small flash memory. > > > > > > > > > > As Greg mentioned: the answer depends on a tradeoff between effort > > and > > > > test > > > > > coverage. > > > > > > > > > > We need to clarify whether we define ostest as a lightweight, > > > > ready-to-run > > > > > test set for quick verification, or as part of the full test suite. > > > > > > > > > > > > > > > The follow is my compiling for cmocka: > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh > sim:citest > > > > -j8* > > > > > Copy files > > > > > Select CONFIG_HOST_LINUX=y > > > > > Refreshing... > > > > > CP: arch/dummy/Kconfig to > > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > > > > > CP: boards/dummy/Kconfig to > > > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > > > > > LN: include/arch to arch/sim/include > > > > > LN: drivers/platform to > > > /home/ligd/platform/mainline/nuttx/drivers/dummy > > > > > LN: arch/sim/src/chip to > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > > > > > LN: arch/sim/src/board to > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > > > > > LN: include/arch/board to > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > > > > > LN: include/arch/chip to > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > > > > > LN: platform/board to > > /home/ligd/platform/mainline/apps/platform/dummy > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/module > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial > > > > > mkkconfig in > > /home/ligd/platform/mainline/apps/interpreters/luamodules > > > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154 > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless > > > > > mkkconfig in /home/ligd/platform/mainline/apps > > > > > Loaded configuration '.config' > > > > > Configuration saved to '.config' > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make* > > > > > Create version.h > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to > > > > > /home/ligd/platform/mainline/apps/platform/dummy > > > > > Downloading: v1.46.0.zip > > > > > ... > > > > > patching file > > > > > > > > > > > > > > > > > > > > testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > > > > > > > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > > > > > LD: nuttx > > > > > Pac SIM with dynamic libs.. > > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1' > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> > > 'sim-pac/libs/libstdc++.so.6' > > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6' > > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6' > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> > 'sim-pac/libs/libgcc_s.so.1' > > > > > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2' > > > > > SIM elf with dynamic libs archive in nuttx.tgz > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > > > > > CHelloWorld: Constructor: mSecret=42 > > > > > nsh: init: open failed: 2 > > > > > > > > > > NuttShell (NSH) NuttX-12.11.0 > > > > > MOTD: username=admin password=Administrator > > > > > nsh> > > > > > nsh> help > > > > > help usage: help [-v] [<cmd>] > > > > > > > > > > . df ifconfig mkfifo readlink > > umount > > > > > [ dmesg ifdown mkrd rm > unset > > > > > ? echo ifup mount rmdir > > uptime > > > > > alias env insmod mv rmmod > > usleep > > > > > unalias exec kill nslookup set > watch > > > > > basename exit pkill pidof sleep xd > > > > > break expr losetup pmconfig source > wait > > > > > cat false ln poweroff test > > > > > cd fdinfo ls quit time > > > > > cp free lsmod printf true > > > > > cmp help mkdir ps truncate > > > > > dirname hexdump mkfatfs pwd uname > > > > > > > > > > Builtin Apps: > > > > > cmocka > > > > > cmocka_driver_block > > > > > cmocka_driver_gpio > > > > > cmocka_driver_pm > > > > > cmocka_driver_pm_runtime > > > > > cmocka_driver_regulator > > > > > > > > > > Matteo Golin <[email protected]> 于2025年12月22日周一 10:15写道： > > > > > > > > > > > I think so. Best to leave OS test alone and slowly replace it > with > > a > > > > new > > > > > > tool that has improvements. The only reason I can see to modify > OS > > > test > > > > > now > > > > > > would just be to have clearer output about what passed/failed. > > > > > > > > > > > > I'll come up with an improved proof of concept for a new tool to > > > share > > > > in > > > > > > the next few days. > > > > > > > > > > > > Matteo > > > > > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <[email protected]> > > wrote: > > > > > > > > > > > > > Small steps with measurable results may be the best approach > > here, > > > to > > > > > > > create some new solution next to existing ostest but not > > replacing > > > it > > > > > > > until ready? :-) > > > > > > > > > > > > > > -- > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > I agree, I think it would take a long time to have proper > > > coverage > > > > of > > > > > > > NuttX > > > > > > > > as a whole. Perhaps even impossible, since Unity is more of a > > > unit > > > > > > > testing > > > > > > > > framework and thus there are definitely test cases for NuttX > > that > > > > > would > > > > > > > be > > > > > > > > outside of its realm of "applicability". There is still a > need > > > for > > > > a > > > > > > > > variety of testing applications and bench-marking > applications > > > for > > > > > > those > > > > > > > > scenarios. > > > > > > > > > > > > > > > > That makes sense, prerequisites would need to be considered. > I > > > > think > > > > > > > > targeting the low-hanging fruit of unit-testable code would > be > > > easy > > > > > in > > > > > > > that > > > > > > > > case, since prerequisites are mostly encoded in the Kconfig > > > > > definitions > > > > > > > > existing or not for features to be tested. Any complex > > behaviour > > > > > > > > (interrupts, multi-threading, etc.) might be out of the realm > > of > > > > > what's > > > > > > > > testable by Unity in the first place. We'll always have OS > test > > > in > > > > > that > > > > > > > > case, but maybe then it's better to do what Alan suggested > and > > at > > > > > least > > > > > > > > make its output more clear in the meantime. I suppose RTOS > > > testing > > > > > is a > > > > > > > > whole field in itself. > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > I think that the answer depends on a tradeoff between > effort > > > and > > > > > test > > > > > > > > > coverage. I could imagine that a proper job could be > > man-years > > > > of > > > > > > > effort. > > > > > > > > > > > > > > > > > > My only concern with a re-design is to be certain that > tests > > > are > > > > > > > > > arranged so that the prerequisites of later test are tested > > > > first. > > > > > > If > > > > > > > > > tests are grouped by, for example, functional area, then > > > failures > > > > > can > > > > > > > be > > > > > > > > > difficult to understand if there is a failure in an > > unverified > > > > test > > > > > > > > > prerequisite. > > > > > > > > > > > > > > > > > > Some of the "tests" are bogus and only exist to provide > some > > > > > minimum > > > > > > > > > confidence that we have the functionality to run the test. > > For > > > > > > > > > example: Can I start tasks? Can I pass parameters? Can I > > > > survive > > > > > a > > > > > > > > > context switch. Some of the tests were intended only to > > > support > > > > > > debug. > > > > > > > > > Some were created only to track down specific bugs. > > > > > > > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote: > > > > > > > > > > LTS being the Long-Term-Stable release of NuttX? > > > > > > > > > > > > > > > > > > > > I'm starting to think that it would be best to create new > > > suite > > > > > of > > > > > > > tests > > > > > > > > > > using Unity, using what's in OS Test as a basis and then > > > > > expanding > > > > > > it > > > > > > > > > > further. Greg has pointed out that OS test is not > intended > > to > > > > be > > > > > a > > > > > > > > > complete > > > > > > > > > > test, so maybe a more solid testing framework base in a > new > > > > form > > > > > > > would be > > > > > > > > > > in order. At that point it would also be good to document > > and > > > > > > educate > > > > > > > > > users > > > > > > > > > > about other test options, since OS test is a current > > > favourite > > > > > for > > > > > > > > > showing > > > > > > > > > > PRs haven't broken anything. Maybe that's not the best > > choice > > > > > > anymore > > > > > > > > > with > > > > > > > > > > this information. > > > > > > > > > > > > > > > > > > > > I don't want to re-invent the wheel of what's already in > > > > > > `testsuite`, > > > > > > > > > but I > > > > > > > > > > think Unity might end up being lighter-weight in the long > > > run? > > > > > And > > > > > > > it is > > > > > > > > > > also not a Xiaomi owned/maintained framework so it may > > > receive > > > > > > better > > > > > > > > > > external support (i.e. more users rely on it?). I tried > to > > > > > compile > > > > > > > the > > > > > > > > > > sim:citest configuration to run the testsuite example and > > see > > > > > what > > > > > > > it was > > > > > > > > > > like, but I was met with a long list of C++ compilation > > > errors > > > > > that > > > > > > > I'd > > > > > > > > > > rather not deal with. There also seems to be zero docs > > about > > > it > > > > > or > > > > > > > cmocka > > > > > > > > > > in our NuttX docs at the moment. > > > > > > > > > > > > > > > > > > > > What do you think? Start a Unity test suite, add some > > > switches > > > > to > > > > > > > choose > > > > > > > > > > which test cases get compiled into the binary, open for > > > > > extensions > > > > > > by > > > > > > > > > users > > > > > > > > > > who know more about different subsystems? This way we can > > > keep > > > > OS > > > > > > > test > > > > > > > > > how > > > > > > > > > > it is for a quick, low-resource sanity check of "board > > > bringup" > > > > > > while > > > > > > > > > > having a dedicated testing application for patches? > Someone > > > > > > changing > > > > > > > file > > > > > > > > > > system stuff can just compile to include file system test > > > cases > > > > > and > > > > > > > run > > > > > > > > > > that suite only as proof that their PR works. The output > is > > > > > really > > > > > > > easy > > > > > > > > > to > > > > > > > > > > glance at to determine pass/fail results. > > > > > > > > > > > > > > > > > > > > Anyways, here's my draft demonstrating how Unity works > with > > > the > > > > > FPU > > > > > > > test > > > > > > > > > > suite from OS test: > > > > > https://github.com/apache/nuttx-apps/pull/3259 > > > > > > > > > > > > > > > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >>> (..) > > > > > > > > > >>> Starting with bits and pieces of something like the LTS > > > might > > > > > > make > > > > > > > more > > > > > > > > > >>> sense other than its licensing problems. Since it > would > > > > never > > > > > be > > > > > > > > > >>> included in a distributed binary, the licensing really > > > should > > > > > not > > > > > > > > > matter > > > > > > > > > >>> much in practice. > > > > > > > > > >> Unless its GPL and some company would like to have their > > own > > > > > > > internal > > > > > > > > > >> bits and pieces.. luckily its MIT and the whole thing is > > > > > > open-source > > > > > > > > > >> anyways :-) > > > > > > > > > >> > > > > > > > > > >> Yes we had many discussions about the LTS and this could > > be > > > > > great > > > > > > > start! > > > > > > > > > >> :-) > > > > > > > > > >> > > > > > > > > > >> -- > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
