Unit testing by itself isn't suitable for small systems. Ostest is something different (system level testing), so running it on small systems makes sense. Running unit tests on small systems is pointless because you won't fit in FLASH either way. You can run the same on large targets (preferably on host: sim, qemu) and achieve the same result.
pon., 22 gru 2025 o 13:35 Alan C. Assis <[email protected]> napisał(a): > Is it too complicated to rewrite the cmocka tests in Unity? > > If cmoka is not feasible for small microcontrollers, like AVRs and others, > our test coverage will be faulty. > > BR, > > Alan > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <[email protected]> wrote: > > > I'm a fan of Unity and have been using it in my projects for years, but I > > think > > we should stick with one testing framework. Currently, Cmocka is used in > > nuttx-apps, and we should stick with it. The only question is how well > > Cmocka > > is suitable for running on small systems? Will ostest still be able to > run > > on small MCUs? > > With Unity this would be quite likely, so maybe an exception for Unity > and > > ostest would > > be acceptable, but using it for other test cases will only cause chaos. > > > > pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> napisał(a): > > > > > Hi Matteo, > > > > > > Good idea ! > > > > > > But it is best to stick with a single testing framework. Currently, > many > > > test cases are using cmocka, so I suggest reusing cmocka for > consistency. > > > > > > ostest is a minimal basic test suite, typically used for CT (continuous > > > testing) of each patch. > > > If we move ostest to cmocka, it would make ostest dependent on a > specific > > > framework before it can be run. > > > And It may increase the code size, which is not very friendly for > > > configurations with small flash memory. > > > > > > As Greg mentioned: the answer depends on a tradeoff between effort and > > test > > > coverage. > > > > > > We need to clarify whether we define ostest as a lightweight, > > ready-to-run > > > test set for quick verification, or as part of the full test suite. > > > > > > > > > The follow is my compiling for cmocka: > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest > > -j8* > > > Copy files > > > Select CONFIG_HOST_LINUX=y > > > Refreshing... > > > CP: arch/dummy/Kconfig to > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > > > CP: boards/dummy/Kconfig to > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > > > LN: include/arch to arch/sim/include > > > LN: drivers/platform to > /home/ligd/platform/mainline/nuttx/drivers/dummy > > > LN: arch/sim/src/chip to > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > > > LN: arch/sim/src/board to > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > > > LN: include/arch/board to > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > > > LN: include/arch/chip to > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > > > LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > > > mkkconfig in /home/ligd/platform/mainline/apps/boot > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto > > > mkkconfig in /home/ligd/platform/mainline/apps/database > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/module > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot > > > mkkconfig in /home/ligd/platform/mainline/apps/examples > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils > > > mkkconfig in /home/ligd/platform/mainline/apps/games > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics > > > mkkconfig in /home/ligd/platform/mainline/apps/industry > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters > > > mkkconfig in /home/ligd/platform/mainline/apps/logging > > > mkkconfig in /home/ligd/platform/mainline/apps/lte > > > mkkconfig in /home/ligd/platform/mainline/apps/math > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr > > > mkkconfig in /home/ligd/platform/mainline/apps/system > > > mkkconfig in /home/ligd/platform/mainline/apps/tee > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx > > > mkkconfig in /home/ligd/platform/mainline/apps/testing > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154 > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless > > > mkkconfig in /home/ligd/platform/mainline/apps > > > Loaded configuration '.config' > > > Configuration saved to '.config' > > > ligd@BAI:~/platform/mainline/nuttx$ > > > *ligd@BAI:~/platform/mainline/nuttx$ make* > > > Create version.h > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to > > > /home/ligd/platform/mainline/apps/platform/dummy > > > Downloading: v1.46.0.zip > > > ... > > > patching file > > > > > > > > > testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > > > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > > > LD: nuttx > > > Pac SIM with dynamic libs.. > > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1' > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> 'sim-pac/libs/libstdc++.so.6' > > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6' > > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6' > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1' > > > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2' > > > SIM elf with dynamic libs archive in nuttx.tgz > > > ligd@BAI:~/platform/mainline/nuttx$ > > > ligd@BAI:~/platform/mainline/nuttx$ > > > ligd@BAI:~/platform/mainline/nuttx$ > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > > > CHelloWorld: Constructor: mSecret=42 > > > nsh: init: open failed: 2 > > > > > > NuttShell (NSH) NuttX-12.11.0 > > > MOTD: username=admin password=Administrator > > > nsh> > > > nsh> help > > > help usage: help [-v] [<cmd>] > > > > > > . df ifconfig mkfifo readlink umount > > > [ dmesg ifdown mkrd rm unset > > > ? echo ifup mount rmdir uptime > > > alias env insmod mv rmmod usleep > > > unalias exec kill nslookup set watch > > > basename exit pkill pidof sleep xd > > > break expr losetup pmconfig source wait > > > cat false ln poweroff test > > > cd fdinfo ls quit time > > > cp free lsmod printf true > > > cmp help mkdir ps truncate > > > dirname hexdump mkfatfs pwd uname > > > > > > Builtin Apps: > > > cmocka > > > cmocka_driver_block > > > cmocka_driver_gpio > > > cmocka_driver_pm > > > cmocka_driver_pm_runtime > > > cmocka_driver_regulator > > > > > > Matteo Golin <[email protected]> 于2025年12月22日周一 10:15写道： > > > > > > > I think so. Best to leave OS test alone and slowly replace it with a > > new > > > > tool that has improvements. The only reason I can see to modify OS > test > > > now > > > > would just be to have clearer output about what passed/failed. > > > > > > > > I'll come up with an improved proof of concept for a new tool to > share > > in > > > > the next few days. > > > > > > > > Matteo > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <[email protected]> wrote: > > > > > > > > > Small steps with measurable results may be the best approach here, > to > > > > > create some new solution next to existing ostest but not replacing > it > > > > > until ready? :-) > > > > > > > > > > -- > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > I agree, I think it would take a long time to have proper > coverage > > of > > > > > NuttX > > > > > > as a whole. Perhaps even impossible, since Unity is more of a > unit > > > > > testing > > > > > > framework and thus there are definitely test cases for NuttX that > > > would > > > > > be > > > > > > outside of its realm of "applicability". There is still a need > for > > a > > > > > > variety of testing applications and bench-marking applications > for > > > > those > > > > > > scenarios. > > > > > > > > > > > > That makes sense, prerequisites would need to be considered. I > > think > > > > > > targeting the low-hanging fruit of unit-testable code would be > easy > > > in > > > > > that > > > > > > case, since prerequisites are mostly encoded in the Kconfig > > > definitions > > > > > > existing or not for features to be tested. Any complex behaviour > > > > > > (interrupts, multi-threading, etc.) might be out of the realm of > > > what's > > > > > > testable by Unity in the first place. We'll always have OS test > in > > > that > > > > > > case, but maybe then it's better to do what Alan suggested and at > > > least > > > > > > make its output more clear in the meantime. I suppose RTOS > testing > > > is a > > > > > > whole field in itself. > > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > I think that the answer depends on a tradeoff between effort > and > > > test > > > > > > > coverage. I could imagine that a proper job could be man-years > > of > > > > > effort. > > > > > > > > > > > > > > My only concern with a re-design is to be certain that tests > are > > > > > > > arranged so that the prerequisites of later test are tested > > first. > > > > If > > > > > > > tests are grouped by, for example, functional area, then > failures > > > can > > > > > be > > > > > > > difficult to understand if there is a failure in an unverified > > test > > > > > > > prerequisite. > > > > > > > > > > > > > > Some of the "tests" are bogus and only exist to provide some > > > minimum > > > > > > > confidence that we have the functionality to run the test. For > > > > > > > example: Can I start tasks? Can I pass parameters? Can I > > survive > > > a > > > > > > > context switch. Some of the tests were intended only to > support > > > > debug. > > > > > > > Some were created only to track down specific bugs. > > > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote: > > > > > > > > LTS being the Long-Term-Stable release of NuttX? > > > > > > > > > > > > > > > > I'm starting to think that it would be best to create new > suite > > > of > > > > > tests > > > > > > > > using Unity, using what's in OS Test as a basis and then > > > expanding > > > > it > > > > > > > > further. Greg has pointed out that OS test is not intended to > > be > > > a > > > > > > > complete > > > > > > > > test, so maybe a more solid testing framework base in a new > > form > > > > > would be > > > > > > > > in order. At that point it would also be good to document and > > > > educate > > > > > > > users > > > > > > > > about other test options, since OS test is a current > favourite > > > for > > > > > > > showing > > > > > > > > PRs haven't broken anything. Maybe that's not the best choice > > > > anymore > > > > > > > with > > > > > > > > this information. > > > > > > > > > > > > > > > > I don't want to re-invent the wheel of what's already in > > > > `testsuite`, > > > > > > > but I > > > > > > > > think Unity might end up being lighter-weight in the long > run? > > > And > > > > > it is > > > > > > > > also not a Xiaomi owned/maintained framework so it may > receive > > > > better > > > > > > > > external support (i.e. more users rely on it?). I tried to > > > compile > > > > > the > > > > > > > > sim:citest configuration to run the testsuite example and see > > > what > > > > > it was > > > > > > > > like, but I was met with a long list of C++ compilation > errors > > > that > > > > > I'd > > > > > > > > rather not deal with. There also seems to be zero docs about > it > > > or > > > > > cmocka > > > > > > > > in our NuttX docs at the moment. > > > > > > > > > > > > > > > > What do you think? Start a Unity test suite, add some > switches > > to > > > > > choose > > > > > > > > which test cases get compiled into the binary, open for > > > extensions > > > > by > > > > > > > users > > > > > > > > who know more about different subsystems? This way we can > keep > > OS > > > > > test > > > > > > > how > > > > > > > > it is for a quick, low-resource sanity check of "board > bringup" > > > > while > > > > > > > > having a dedicated testing application for patches? Someone > > > > changing > > > > > file > > > > > > > > system stuff can just compile to include file system test > cases > > > and > > > > > run > > > > > > > > that suite only as proof that their PR works. The output is > > > really > > > > > easy > > > > > > > to > > > > > > > > glance at to determine pass/fail results. > > > > > > > > > > > > > > > > Anyways, here's my draft demonstrating how Unity works with > the > > > FPU > > > > > test > > > > > > > > suite from OS test: > > > https://github.com/apache/nuttx-apps/pull/3259 > > > > > > > > > > > > > > > > Matteo > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > >>> (..) > > > > > > > >>> Starting with bits and pieces of something like the LTS > might > > > > make > > > > > more > > > > > > > >>> sense other than its licensing problems. Since it would > > never > > > be > > > > > > > >>> included in a distributed binary, the licensing really > should > > > not > > > > > > > matter > > > > > > > >>> much in practice. > > > > > > > >> Unless its GPL and some company would like to have their own > > > > > internal > > > > > > > >> bits and pieces.. luckily its MIT and the whole thing is > > > > open-source > > > > > > > >> anyways :-) > > > > > > > >> > > > > > > > >> Yes we had many discussions about the LTS and this could be > > > great > > > > > start! > > > > > > > >> :-) > > > > > > > >> > > > > > > > >> -- > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >
