I wonder if the 8 bit architectures use smaller integer sizes for things like the timespec structs. I wonder if there are any other parts of NuttX that would complain during compilation seeing that ostest seems to make the assumption that tv_nsec is big enough for that operation. I guess its infrequent that things are tested on 8-bit MCUs now.
As someone who is regularly developing for AVR, do you think it would be a benefit to have a test suite that is able to run fully on AVR itself? Even if only a couple tests can be included in an image at once due to flash size limits? Merry Christmas, Matteo On Thu, Dec 25, 2025 at 8:18 AM <[email protected]> wrote: > I was wondering if ostest is even supposed to work on AVR so I gave it a > try... it didn't do much, the only result I got when I ran ostest from > NSH was: > > stdio_test: write fd=1 > stdio_test: write fd=2 > > And that's it. > > (There may be something wrong with printf when the application gets > executed through NSH though - just noticed that my other testing apps > also print nothing when executed from NSH while being able to printf > when executed as init entrypoint. Will try to update apps to something > more recent but if anyone has any idea what may be causing this, it > would be appreciated.) > > Anyway, the compilation also throws a lot of warnings, for example > > semtimed.c:87:32: warning: integer overflow in expression of type ‘int’ > results in ‘16960’ [-Woverflow] > 87 | tp->tv_nsec >= 1000 * 1000 * 1000) > > and there are also warnings regarding to size mismatch in > pointer/integer conversion. > > So it may be that ostest doesn't work well for AVR (and probably for > other 8 bit MCUs) already. It also already barely fits the flash of the > largest AVR DA chip. > > (Not suggesting to make it even more faulty, just mentioning it FYI.) > > On 2025-12-22 13:35, Alan C. Assis wrote: > > Is it too complicated to rewrite the cmocka tests in Unity? > > > > If cmoka is not feasible for small microcontrollers, like AVRs and > > others, > > our test coverage will be faulty. > > > > BR, > > > > Alan > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <[email protected]> > > wrote: > > > >> I'm a fan of Unity and have been using it in my projects for years, > >> but I > >> think > >> we should stick with one testing framework. Currently, Cmocka is used > >> in > >> nuttx-apps, and we should stick with it. The only question is how well > >> Cmocka > >> is suitable for running on small systems? Will ostest still be able to > >> run > >> on small MCUs? > >> With Unity this would be quite likely, so maybe an exception for Unity > >> and > >> ostest would > >> be acceptable, but using it for other test cases will only cause > >> chaos. > >> > >> pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> napisał(a): > >> > >> > Hi Matteo, > >> > > >> > Good idea ! > >> > > >> > But it is best to stick with a single testing framework. Currently, > many > >> > test cases are using cmocka, so I suggest reusing cmocka for > consistency. > >> > > >> > ostest is a minimal basic test suite, typically used for CT > (continuous > >> > testing) of each patch. > >> > If we move ostest to cmocka, it would make ostest dependent on a > specific > >> > framework before it can be run. > >> > And It may increase the code size, which is not very friendly for > >> > configurations with small flash memory. > >> > > >> > As Greg mentioned: the answer depends on a tradeoff between effort and > >> test > >> > coverage. > >> > > >> > We need to clarify whether we define ostest as a lightweight, > >> ready-to-run > >> > test set for quick verification, or as part of the full test suite. > >> > > >> > > >> > The follow is my compiling for cmocka: > >> > > >> > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest > >> -j8* > >> > Copy files > >> > Select CONFIG_HOST_LINUX=y > >> > Refreshing... > >> > CP: arch/dummy/Kconfig to > >> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > >> > CP: boards/dummy/Kconfig to > >> > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > >> > LN: include/arch to arch/sim/include > >> > LN: drivers/platform to > /home/ligd/platform/mainline/nuttx/drivers/dummy > >> > LN: arch/sim/src/chip to > >> > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > >> > LN: arch/sim/src/board to > >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > >> > LN: include/arch/board to > >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > >> > LN: include/arch/chip to > >> > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > >> > LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy > >> > mkkconfig in /home/ligd/platform/mainline/apps/audioutils > >> > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > >> > mkkconfig in /home/ligd/platform/mainline/apps/boot > >> > mkkconfig in /home/ligd/platform/mainline/apps/canutils > >> > mkkconfig in /home/ligd/platform/mainline/apps/crypto > >> > mkkconfig in /home/ligd/platform/mainline/apps/database > >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust > >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest > >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/module > >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot > >> > mkkconfig in /home/ligd/platform/mainline/apps/examples > >> > mkkconfig in /home/ligd/platform/mainline/apps/fsutils > >> > mkkconfig in /home/ligd/platform/mainline/apps/games > >> > mkkconfig in /home/ligd/platform/mainline/apps/graphics > >> > mkkconfig in /home/ligd/platform/mainline/apps/industry > >> > mkkconfig in /home/ligd/platform/mainline/apps/inertial > >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules > >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters > >> > mkkconfig in /home/ligd/platform/mainline/apps/logging > >> > mkkconfig in /home/ligd/platform/mainline/apps/lte > >> > mkkconfig in /home/ligd/platform/mainline/apps/math > >> > mkkconfig in /home/ligd/platform/mainline/apps/mlearning > >> > mkkconfig in /home/ligd/platform/mainline/apps/netutils > >> > mkkconfig in /home/ligd/platform/mainline/apps/sdr > >> > mkkconfig in /home/ligd/platform/mainline/apps/system > >> > mkkconfig in /home/ligd/platform/mainline/apps/tee > >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers > >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched > >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc > >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch > >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx > >> > mkkconfig in /home/ligd/platform/mainline/apps/testing > >> > mkkconfig in /home/ligd/platform/mainline/apps/videoutils > >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth > >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154 > >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless > >> > mkkconfig in /home/ligd/platform/mainline/apps > >> > Loaded configuration '.config' > >> > Configuration saved to '.config' > >> > ligd@BAI:~/platform/mainline/nuttx$ > >> > *ligd@BAI:~/platform/mainline/nuttx$ make* > >> > Create version.h > >> > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to > >> > /home/ligd/platform/mainline/apps/platform/dummy > >> > Downloading: v1.46.0.zip > >> > ... > >> > patching file > >> > > >> > > >> > testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > >> > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > >> > > >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > >> > LD: nuttx > >> > Pac SIM with dynamic libs.. > >> > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1' > >> > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> > 'sim-pac/libs/libstdc++.so.6' > >> > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6' > >> > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6' > >> > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1' > >> > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2' > >> > SIM elf with dynamic libs archive in nuttx.tgz > >> > ligd@BAI:~/platform/mainline/nuttx$ > >> > ligd@BAI:~/platform/mainline/nuttx$ > >> > ligd@BAI:~/platform/mainline/nuttx$ > >> > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > >> > CHelloWorld: Constructor: mSecret=42 > >> > nsh: init: open failed: 2 > >> > > >> > NuttShell (NSH) NuttX-12.11.0 > >> > MOTD: username=admin password=Administrator > >> > nsh> > >> > nsh> help > >> > help usage: help [-v] [<cmd>] > >> > > >> > . df ifconfig mkfifo readlink umount > >> > [ dmesg ifdown mkrd rm unset > >> > ? echo ifup mount rmdir uptime > >> > alias env insmod mv rmmod usleep > >> > unalias exec kill nslookup set watch > >> > basename exit pkill pidof sleep xd > >> > break expr losetup pmconfig source wait > >> > cat false ln poweroff test > >> > cd fdinfo ls quit time > >> > cp free lsmod printf true > >> > cmp help mkdir ps truncate > >> > dirname hexdump mkfatfs pwd uname > >> > > >> > Builtin Apps: > >> > cmocka > >> > cmocka_driver_block > >> > cmocka_driver_gpio > >> > cmocka_driver_pm > >> > cmocka_driver_pm_runtime > >> > cmocka_driver_regulator > >> > > >> > Matteo Golin <[email protected]> 于2025年12月22日周一 10:15写道： > >> > > >> > > I think so. Best to leave OS test alone and slowly replace it with a > >> new > >> > > tool that has improvements. The only reason I can see to modify OS > test > >> > now > >> > > would just be to have clearer output about what passed/failed. > >> > > > >> > > I'll come up with an improved proof of concept for a new tool to > share > >> in > >> > > the next few days. > >> > > > >> > > Matteo > >> > > > >> > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <[email protected]> wrote: > >> > > > >> > > > Small steps with measurable results may be the best approach > here, to > >> > > > create some new solution next to existing ostest but not > replacing it > >> > > > until ready? :-) > >> > > > > >> > > > -- > >> > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > >> > > > > >> > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > [email protected] > >> > > >> > > > wrote: > >> > > > > > >> > > > > I agree, I think it would take a long time to have proper > coverage > >> of > >> > > > NuttX > >> > > > > as a whole. Perhaps even impossible, since Unity is more of a > unit > >> > > > testing > >> > > > > framework and thus there are definitely test cases for NuttX > that > >> > would > >> > > > be > >> > > > > outside of its realm of "applicability". There is still a need > for > >> a > >> > > > > variety of testing applications and bench-marking applications > for > >> > > those > >> > > > > scenarios. > >> > > > > > >> > > > > That makes sense, prerequisites would need to be considered. I > >> think > >> > > > > targeting the low-hanging fruit of unit-testable code would be > easy > >> > in > >> > > > that > >> > > > > case, since prerequisites are mostly encoded in the Kconfig > >> > definitions > >> > > > > existing or not for features to be tested. Any complex behaviour > >> > > > > (interrupts, multi-threading, etc.) might be out of the realm of > >> > what's > >> > > > > testable by Unity in the first place. We'll always have OS test > in > >> > that > >> > > > > case, but maybe then it's better to do what Alan suggested and > at > >> > least > >> > > > > make its output more clear in the meantime. I suppose RTOS > testing > >> > is a > >> > > > > whole field in itself. > >> > > > > > >> > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > [email protected]> > >> > > > wrote: > >> > > > > > >> > > > > > I think that the answer depends on a tradeoff between effort > and > >> > test > >> > > > > > coverage. I could imagine that a proper job could be > man-years > >> of > >> > > > effort. > >> > > > > > > >> > > > > > My only concern with a re-design is to be certain that tests > are > >> > > > > > arranged so that the prerequisites of later test are tested > >> first. > >> > > If > >> > > > > > tests are grouped by, for example, functional area, then > failures > >> > can > >> > > > be > >> > > > > > difficult to understand if there is a failure in an unverified > >> test > >> > > > > > prerequisite. > >> > > > > > > >> > > > > > Some of the "tests" are bogus and only exist to provide some > >> > minimum > >> > > > > > confidence that we have the functionality to run the test. > For > >> > > > > > example: Can I start tasks? Can I pass parameters? Can I > >> survive > >> > a > >> > > > > > context switch. Some of the tests were intended only to > support > >> > > debug. > >> > > > > > Some were created only to track down specific bugs. > >> > > > > > > >> > > > > > On 12/21/25 18:39, Matteo Golin wrote: > >> > > > > > > LTS being the Long-Term-Stable release of NuttX? > >> > > > > > > > >> > > > > > > I'm starting to think that it would be best to create new > suite > >> > of > >> > > > tests > >> > > > > > > using Unity, using what's in OS Test as a basis and then > >> > expanding > >> > > it > >> > > > > > > further. Greg has pointed out that OS test is not intended > to > >> be > >> > a > >> > > > > > complete > >> > > > > > > test, so maybe a more solid testing framework base in a new > >> form > >> > > > would be > >> > > > > > > in order. At that point it would also be good to document > and > >> > > educate > >> > > > > > users > >> > > > > > > about other test options, since OS test is a current > favourite > >> > for > >> > > > > > showing > >> > > > > > > PRs haven't broken anything. Maybe that's not the best > choice > >> > > anymore > >> > > > > > with > >> > > > > > > this information. > >> > > > > > > > >> > > > > > > I don't want to re-invent the wheel of what's already in > >> > > `testsuite`, > >> > > > > > but I > >> > > > > > > think Unity might end up being lighter-weight in the long > run? > >> > And > >> > > > it is > >> > > > > > > also not a Xiaomi owned/maintained framework so it may > receive > >> > > better > >> > > > > > > external support (i.e. more users rely on it?). I tried to > >> > compile > >> > > > the > >> > > > > > > sim:citest configuration to run the testsuite example and > see > >> > what > >> > > > it was > >> > > > > > > like, but I was met with a long list of C++ compilation > errors > >> > that > >> > > > I'd > >> > > > > > > rather not deal with. There also seems to be zero docs > about it > >> > or > >> > > > cmocka > >> > > > > > > in our NuttX docs at the moment. > >> > > > > > > > >> > > > > > > What do you think? Start a Unity test suite, add some > switches > >> to > >> > > > choose > >> > > > > > > which test cases get compiled into the binary, open for > >> > extensions > >> > > by > >> > > > > > users > >> > > > > > > who know more about different subsystems? This way we can > keep > >> OS > >> > > > test > >> > > > > > how > >> > > > > > > it is for a quick, low-resource sanity check of "board > bringup" > >> > > while > >> > > > > > > having a dedicated testing application for patches? Someone > >> > > changing > >> > > > file > >> > > > > > > system stuff can just compile to include file system test > cases > >> > and > >> > > > run > >> > > > > > > that suite only as proof that their PR works. The output is > >> > really > >> > > > easy > >> > > > > > to > >> > > > > > > glance at to determine pass/fail results. > >> > > > > > > > >> > > > > > > Anyways, here's my draft demonstrating how Unity works with > the > >> > FPU > >> > > > test > >> > > > > > > suite from OS test: > >> > https://github.com/apache/nuttx-apps/pull/3259 > >> > > > > > > > >> > > > > > > Matteo > >> > > > > > > > >> > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > [email protected]> > >> > > > wrote: > >> > > > > > > > >> > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > >> > [email protected] > >> > > > > >> > > > > > wrote: > >> > > > > > >>> (..) > >> > > > > > >>> Starting with bits and pieces of something like the LTS > might > >> > > make > >> > > > more > >> > > > > > >>> sense other than its licensing problems. Since it would > >> never > >> > be > >> > > > > > >>> included in a distributed binary, the licensing really > should > >> > not > >> > > > > > matter > >> > > > > > >>> much in practice. > >> > > > > > >> Unless its GPL and some company would like to have their > own > >> > > > internal > >> > > > > > >> bits and pieces.. luckily its MIT and the whole thing is > >> > > open-source > >> > > > > > >> anyways :-) > >> > > > > > >> > >> > > > > > >> Yes we had many discussions about the LTS and this could be > >> > great > >> > > > start! > >> > > > > > >> :-) > >> > > > > > >> > >> > > > > > >> -- > >> > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > >> > > > > > >> > >> > > > > > > >> > > > > >> > > > >> > > >> >
