Yes, I agree! We need data size from each solution for comparison, this is the first step.
Maybe the conversion could be facilitated if we create equivalent functions like those used in the apps/testing/testsuites: cmocka_run_group_tests -> unity_run_group_tests cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown etc... This way we avoid end up with two unit test framework, we already have the Makefile vs CMake precedent! :-D On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <[email protected]> wrote: > The discussion about small systems doesn't make much sense without > data comparing resources of the same test cases for cmock and unity. > > But let's assume that Unity is better. Should we use two frameworks > to implement unit tests for OS? Or choose one framework? If we choose only > Unity, who > will migrate the current cmocka tests to Unity? Remember that cmoka tests > are used in CI, so all CI tools must be also migrated. > Or should we use two different frameworks to test different things and make > even > more mess in apps/testing as it is now? > > No one is forcing anyone to do anything. This is an open-source project, > everyone > should do what they feel is right. But whether the change should be merged > is > a different question. In this case, it's a very important question, because > unit testing > isn't just about test cases. It's about the entire infrastructure that > runs, captures and > handles test results later. Writing the test cases is the least of the > problems. > The infrastructure for cmoka testing is already in place, the > infrastructure for Unity > testing isn't. > > pon., 22 gru 2025 o 14:16 Alan C. Assis <[email protected]> napisał(a): > > > Mateusz, > > but some chips like AVR DA have big flash, but not so much RAM. > > > > It is better to have some possible way to test on a small system, using > > cmocka we are removing this option from the table. > > > > I liked the test Matteo did with Unity and I think we should let him > > continue. Forcing him to migrate to cmocka just because there's a test > > suite from Xiaomi doesn't seem prudent to me. > > > > This is the advantage of an open mind open-source project: "the > possibility > > to test new ideas", if someone had said to Michal Lenc he couldn't create > > NXBoot because where is already MCUBoot integrated on NuttX, we didn't > have > > the option to use NXBoot today. ;-) > > > > BR, > > > > Alan > > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <[email protected]> > wrote: > > > > > Unit testing by itself isn't suitable for small systems. Ostest is > > > something > > > different (system level testing), so running it on small systems makes > > > sense. > > > Running unit tests on small systems is pointless because you won't fit > in > > > FLASH > > > either way. You can run the same on large targets (preferably on host: > > sim, > > > qemu) > > > and achieve the same result. > > > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <[email protected]> > napisał(a): > > > > > > > Is it too complicated to rewrite the cmocka tests in Unity? > > > > > > > > If cmoka is not feasible for small microcontrollers, like AVRs and > > > others, > > > > our test coverage will be faulty. > > > > > > > > BR, > > > > > > > > Alan > > > > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <[email protected]> > > wrote: > > > > > > > > > I'm a fan of Unity and have been using it in my projects for years, > > > but I > > > > > think > > > > > we should stick with one testing framework. Currently, Cmocka is > used > > > in > > > > > nuttx-apps, and we should stick with it. The only question is how > > well > > > > > Cmocka > > > > > is suitable for running on small systems? Will ostest still be able > > to > > > > run > > > > > on small MCUs? > > > > > With Unity this would be quite likely, so maybe an exception for > > Unity > > > > and > > > > > ostest would > > > > > be acceptable, but using it for other test cases will only cause > > chaos. > > > > > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> > napisał(a): > > > > > > > > > > > Hi Matteo, > > > > > > > > > > > > Good idea ! > > > > > > > > > > > > But it is best to stick with a single testing framework. > Currently, > > > > many > > > > > > test cases are using cmocka, so I suggest reusing cmocka for > > > > consistency. > > > > > > > > > > > > ostest is a minimal basic test suite, typically used for CT > > > (continuous > > > > > > testing) of each patch. > > > > > > If we move ostest to cmocka, it would make ostest dependent on a > > > > specific > > > > > > framework before it can be run. > > > > > > And It may increase the code size, which is not very friendly for > > > > > > configurations with small flash memory. > > > > > > > > > > > > As Greg mentioned: the answer depends on a tradeoff between > effort > > > and > > > > > test > > > > > > coverage. > > > > > > > > > > > > We need to clarify whether we define ostest as a lightweight, > > > > > ready-to-run > > > > > > test set for quick verification, or as part of the full test > suite. > > > > > > > > > > > > > > > > > > The follow is my compiling for cmocka: > > > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh > > sim:citest > > > > > -j8* > > > > > > Copy files > > > > > > Select CONFIG_HOST_LINUX=y > > > > > > Refreshing... > > > > > > CP: arch/dummy/Kconfig to > > > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > > > > > > CP: boards/dummy/Kconfig to > > > > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > > > > > > LN: include/arch to arch/sim/include > > > > > > LN: drivers/platform to > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy > > > > > > LN: arch/sim/src/chip to > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > > > > > > LN: arch/sim/src/board to > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > > > > > > LN: include/arch/board to > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > > > > > > LN: include/arch/chip to > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > > > > > > LN: platform/board to > > > /home/ligd/platform/mainline/apps/platform/dummy > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/module > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial > > > > > > mkkconfig in > > > /home/ligd/platform/mainline/apps/interpreters/luamodules > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth > > > > > > mkkconfig in > /home/ligd/platform/mainline/apps/wireless/ieee802154 > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless > > > > > > mkkconfig in /home/ligd/platform/mainline/apps > > > > > > Loaded configuration '.config' > > > > > > Configuration saved to '.config' > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make* > > > > > > Create version.h > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to > > > > > > /home/ligd/platform/mainline/apps/platform/dummy > > > > > > Downloading: v1.46.0.zip > > > > > > ... > > > > > > patching file > > > > > > > > > > > > > > > > > > > > > > > > > > > testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > > > > > > > > > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > > > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > > > > > > LD: nuttx > > > > > > Pac SIM with dynamic libs.. > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1' > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> > > > 'sim-pac/libs/libstdc++.so.6' > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6' > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6' > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> > > 'sim-pac/libs/libgcc_s.so.1' > > > > > > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2' > > > > > > SIM elf with dynamic libs archive in nuttx.tgz > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > > > > > > CHelloWorld: Constructor: mSecret=42 > > > > > > nsh: init: open failed: 2 > > > > > > > > > > > > NuttShell (NSH) NuttX-12.11.0 > > > > > > MOTD: username=admin password=Administrator > > > > > > nsh> > > > > > > nsh> help > > > > > > help usage: help [-v] [<cmd>] > > > > > > > > > > > > . df ifconfig mkfifo readlink > > > umount > > > > > > [ dmesg ifdown mkrd rm > > unset > > > > > > ? echo ifup mount rmdir > > > uptime > > > > > > alias env insmod mv rmmod > > > usleep > > > > > > unalias exec kill nslookup set > > watch > > > > > > basename exit pkill pidof sleep > xd > > > > > > break expr losetup pmconfig source > > wait > > > > > > cat false ln poweroff test > > > > > > cd fdinfo ls quit time > > > > > > cp free lsmod printf true > > > > > > cmp help mkdir ps truncate > > > > > > dirname hexdump mkfatfs pwd uname > > > > > > > > > > > > Builtin Apps: > > > > > > cmocka > > > > > > cmocka_driver_block > > > > > > cmocka_driver_gpio > > > > > > cmocka_driver_pm > > > > > > cmocka_driver_pm_runtime > > > > > > cmocka_driver_regulator > > > > > > > > > > > > Matteo Golin <[email protected]> 于2025年12月22日周一 10:15写道： > > > > > > > > > > > > > I think so. Best to leave OS test alone and slowly replace it > > with > > > a > > > > > new > > > > > > > tool that has improvements. The only reason I can see to modify > > OS > > > > test > > > > > > now > > > > > > > would just be to have clearer output about what passed/failed. > > > > > > > > > > > > > > I'll come up with an improved proof of concept for a new tool > to > > > > share > > > > > in > > > > > > > the next few days. > > > > > > > > > > > > > > Matteo > > > > > > > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <[email protected]> > > > wrote: > > > > > > > > > > > > > > > Small steps with measurable results may be the best approach > > > here, > > > > to > > > > > > > > create some new solution next to existing ostest but not > > > replacing > > > > it > > > > > > > > until ready? :-) > > > > > > > > > > > > > > > > -- > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > I agree, I think it would take a long time to have proper > > > > coverage > > > > > of > > > > > > > > NuttX > > > > > > > > > as a whole. Perhaps even impossible, since Unity is more > of a > > > > unit > > > > > > > > testing > > > > > > > > > framework and thus there are definitely test cases for > NuttX > > > that > > > > > > would > > > > > > > > be > > > > > > > > > outside of its realm of "applicability". There is still a > > need > > > > for > > > > > a > > > > > > > > > variety of testing applications and bench-marking > > applications > > > > for > > > > > > > those > > > > > > > > > scenarios. > > > > > > > > > > > > > > > > > > That makes sense, prerequisites would need to be > considered. > > I > > > > > think > > > > > > > > > targeting the low-hanging fruit of unit-testable code would > > be > > > > easy > > > > > > in > > > > > > > > that > > > > > > > > > case, since prerequisites are mostly encoded in the Kconfig > > > > > > definitions > > > > > > > > > existing or not for features to be tested. Any complex > > > behaviour > > > > > > > > > (interrupts, multi-threading, etc.) might be out of the > realm > > > of > > > > > > what's > > > > > > > > > testable by Unity in the first place. We'll always have OS > > test > > > > in > > > > > > that > > > > > > > > > case, but maybe then it's better to do what Alan suggested > > and > > > at > > > > > > least > > > > > > > > > make its output more clear in the meantime. I suppose RTOS > > > > testing > > > > > > is a > > > > > > > > > whole field in itself. > > > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I think that the answer depends on a tradeoff between > > effort > > > > and > > > > > > test > > > > > > > > > > coverage. I could imagine that a proper job could be > > > man-years > > > > > of > > > > > > > > effort. > > > > > > > > > > > > > > > > > > > > My only concern with a re-design is to be certain that > > tests > > > > are > > > > > > > > > > arranged so that the prerequisites of later test are > tested > > > > > first. > > > > > > > If > > > > > > > > > > tests are grouped by, for example, functional area, then > > > > failures > > > > > > can > > > > > > > > be > > > > > > > > > > difficult to understand if there is a failure in an > > > unverified > > > > > test > > > > > > > > > > prerequisite. > > > > > > > > > > > > > > > > > > > > Some of the "tests" are bogus and only exist to provide > > some > > > > > > minimum > > > > > > > > > > confidence that we have the functionality to run the > test. > > > For > > > > > > > > > > example: Can I start tasks? Can I pass parameters? Can > I > > > > > survive > > > > > > a > > > > > > > > > > context switch. Some of the tests were intended only to > > > > support > > > > > > > debug. > > > > > > > > > > Some were created only to track down specific bugs. > > > > > > > > > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote: > > > > > > > > > > > LTS being the Long-Term-Stable release of NuttX? > > > > > > > > > > > > > > > > > > > > > > I'm starting to think that it would be best to create > new > > > > suite > > > > > > of > > > > > > > > tests > > > > > > > > > > > using Unity, using what's in OS Test as a basis and > then > > > > > > expanding > > > > > > > it > > > > > > > > > > > further. Greg has pointed out that OS test is not > > intended > > > to > > > > > be > > > > > > a > > > > > > > > > > complete > > > > > > > > > > > test, so maybe a more solid testing framework base in a > > new > > > > > form > > > > > > > > would be > > > > > > > > > > > in order. At that point it would also be good to > document > > > and > > > > > > > educate > > > > > > > > > > users > > > > > > > > > > > about other test options, since OS test is a current > > > > favourite > > > > > > for > > > > > > > > > > showing > > > > > > > > > > > PRs haven't broken anything. Maybe that's not the best > > > choice > > > > > > > anymore > > > > > > > > > > with > > > > > > > > > > > this information. > > > > > > > > > > > > > > > > > > > > > > I don't want to re-invent the wheel of what's already > in > > > > > > > `testsuite`, > > > > > > > > > > but I > > > > > > > > > > > think Unity might end up being lighter-weight in the > long > > > > run? > > > > > > And > > > > > > > > it is > > > > > > > > > > > also not a Xiaomi owned/maintained framework so it may > > > > receive > > > > > > > better > > > > > > > > > > > external support (i.e. more users rely on it?). I tried > > to > > > > > > compile > > > > > > > > the > > > > > > > > > > > sim:citest configuration to run the testsuite example > and > > > see > > > > > > what > > > > > > > > it was > > > > > > > > > > > like, but I was met with a long list of C++ compilation > > > > errors > > > > > > that > > > > > > > > I'd > > > > > > > > > > > rather not deal with. There also seems to be zero docs > > > about > > > > it > > > > > > or > > > > > > > > cmocka > > > > > > > > > > > in our NuttX docs at the moment. > > > > > > > > > > > > > > > > > > > > > > What do you think? Start a Unity test suite, add some > > > > switches > > > > > to > > > > > > > > choose > > > > > > > > > > > which test cases get compiled into the binary, open for > > > > > > extensions > > > > > > > by > > > > > > > > > > users > > > > > > > > > > > who know more about different subsystems? This way we > can > > > > keep > > > > > OS > > > > > > > > test > > > > > > > > > > how > > > > > > > > > > > it is for a quick, low-resource sanity check of "board > > > > bringup" > > > > > > > while > > > > > > > > > > > having a dedicated testing application for patches? > > Someone > > > > > > > changing > > > > > > > > file > > > > > > > > > > > system stuff can just compile to include file system > test > > > > cases > > > > > > and > > > > > > > > run > > > > > > > > > > > that suite only as proof that their PR works. The > output > > is > > > > > > really > > > > > > > > easy > > > > > > > > > > to > > > > > > > > > > > glance at to determine pass/fail results. > > > > > > > > > > > > > > > > > > > > > > Anyways, here's my draft demonstrating how Unity works > > with > > > > the > > > > > > FPU > > > > > > > > test > > > > > > > > > > > suite from OS test: > > > > > > https://github.com/apache/nuttx-apps/pull/3259 > > > > > > > > > > > > > > > > > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >>> (..) > > > > > > > > > > >>> Starting with bits and pieces of something like the > LTS > > > > might > > > > > > > make > > > > > > > > more > > > > > > > > > > >>> sense other than its licensing problems. Since it > > would > > > > > never > > > > > > be > > > > > > > > > > >>> included in a distributed binary, the licensing > really > > > > should > > > > > > not > > > > > > > > > > matter > > > > > > > > > > >>> much in practice. > > > > > > > > > > >> Unless its GPL and some company would like to have > their > > > own > > > > > > > > internal > > > > > > > > > > >> bits and pieces.. luckily its MIT and the whole thing > is > > > > > > > open-source > > > > > > > > > > >> anyways :-) > > > > > > > > > > >> > > > > > > > > > > >> Yes we had many discussions about the LTS and this > could > > > be > > > > > > great > > > > > > > > start! > > > > > > > > > > >> :-) > > > > > > > > > > >> > > > > > > > > > > >> -- > > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
