On the other hand AVR is still very popular and widely available so it could be NuttX ostest/benchmark for 8-bit platforms.. I have some small/medium/big AVR MCUs somewhere around so I could find them and help in testing/verification :-)
-- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info On Fri, Dec 26, 2025 at 12:06 PM <[email protected]> wrote: > > With avr-gcc, int is int16_t. Numeric literals are treated as int so > every compile-time calculation larger than 16 bits gets truncated unless > one of the operands is marked as longer type. tv_nsec is big enough to > fit the value (sizeof(long) == 4) but the assignment to it only happens > after the 16 bit truncation. Timing constants section in > include/nuttx/clock.h is an example of what's needed to make such > assignments to behave correctly. > > As for the test suite - it would be nice to have of course but > considering that (I guess) there are not very many people who use NuttX > on AVR, the time spent on making that happen would probably be better > used on something with larger reach. > > (On a related note - there are other similar pitfalls when it comes to > NuttX on AVR and probably on other 8 bit MCUs in general. For example, > any code that assumes that arithmetical/logical operations on int > operand are atomic is faulty in this setting. You can see commit > a3f8b55143ad for possible implications of this. As far as I can see, > struct circbuf_s has similar issue and I wonder if something like this > can be found in core code like the scheduler. Because if so, then having > tests is nice but not that useful since they should always fail under > the circumstances.) > > On 2025-12-25 18:33, Matteo Golin wrote: > > I wonder if the 8 bit architectures use smaller integer sizes for > > things > > like the timespec structs. I wonder if there are any other parts of > > NuttX > > that would complain during compilation seeing that ostest seems to make > > the > > assumption that tv_nsec is big enough for that operation. I guess its > > infrequent that things are tested on 8-bit MCUs now. > > > > As someone who is regularly developing for AVR, do you think it would > > be a > > benefit to have a test suite that is able to run fully on AVR itself? > > Even > > if only a couple tests can be included in an image at once due to flash > > size limits? > > > > Merry Christmas, > > Matteo > > > > On Thu, Dec 25, 2025 at 8:18 AM <[email protected]> wrote: > > > >> I was wondering if ostest is even supposed to work on AVR so I gave it > >> a > >> try... it didn't do much, the only result I got when I ran ostest from > >> NSH was: > >> > >> stdio_test: write fd=1 > >> stdio_test: write fd=2 > >> > >> And that's it. > >> > >> (There may be something wrong with printf when the application gets > >> executed through NSH though - just noticed that my other testing apps > >> also print nothing when executed from NSH while being able to printf > >> when executed as init entrypoint. Will try to update apps to something > >> more recent but if anyone has any idea what may be causing this, it > >> would be appreciated.) > >> > >> Anyway, the compilation also throws a lot of warnings, for example > >> > >> semtimed.c:87:32: warning: integer overflow in expression of type > >> ‘int’ > >> results in ‘16960’ [-Woverflow] > >> 87 | tp->tv_nsec >= 1000 * 1000 * 1000) > >> > >> and there are also warnings regarding to size mismatch in > >> pointer/integer conversion. > >> > >> So it may be that ostest doesn't work well for AVR (and probably for > >> other 8 bit MCUs) already. It also already barely fits the flash of > >> the > >> largest AVR DA chip. > >> > >> (Not suggesting to make it even more faulty, just mentioning it FYI.) > >> > >> On 2025-12-22 13:35, Alan C. Assis wrote: > >> > Is it too complicated to rewrite the cmocka tests in Unity? > >> > > >> > If cmoka is not feasible for small microcontrollers, like AVRs and > >> > others, > >> > our test coverage will be faulty. > >> > > >> > BR, > >> > > >> > Alan > >> > > >> > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <[email protected]> > >> > wrote: > >> > > >> >> I'm a fan of Unity and have been using it in my projects for years, > >> >> but I > >> >> think > >> >> we should stick with one testing framework. Currently, Cmocka is used > >> >> in > >> >> nuttx-apps, and we should stick with it. The only question is how well > >> >> Cmocka > >> >> is suitable for running on small systems? Will ostest still be able to > >> >> run > >> >> on small MCUs? > >> >> With Unity this would be quite likely, so maybe an exception for Unity > >> >> and > >> >> ostest would > >> >> be acceptable, but using it for other test cases will only cause > >> >> chaos. > >> >> > >> >> pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> napisał(a): > >> >> > >> >> > Hi Matteo, > >> >> > > >> >> > Good idea ! > >> >> > > >> >> > But it is best to stick with a single testing framework. Currently, > >> many > >> >> > test cases are using cmocka, so I suggest reusing cmocka for > >> consistency. > >> >> > > >> >> > ostest is a minimal basic test suite, typically used for CT > >> (continuous > >> >> > testing) of each patch. > >> >> > If we move ostest to cmocka, it would make ostest dependent on a > >> specific > >> >> > framework before it can be run. > >> >> > And It may increase the code size, which is not very friendly for > >> >> > configurations with small flash memory. > >> >> > > >> >> > As Greg mentioned: the answer depends on a tradeoff between effort and > >> >> test > >> >> > coverage. > >> >> > > >> >> > We need to clarify whether we define ostest as a lightweight, > >> >> ready-to-run > >> >> > test set for quick verification, or as part of the full test suite. > >> >> > > >> >> > > >> >> > The follow is my compiling for cmocka: > >> >> > > >> >> > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest > >> >> -j8* > >> >> > Copy files > >> >> > Select CONFIG_HOST_LINUX=y > >> >> > Refreshing... > >> >> > CP: arch/dummy/Kconfig to > >> >> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > >> >> > CP: boards/dummy/Kconfig to > >> >> > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > >> >> > LN: include/arch to arch/sim/include > >> >> > LN: drivers/platform to > >> /home/ligd/platform/mainline/nuttx/drivers/dummy > >> >> > LN: arch/sim/src/chip to > >> >> > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > >> >> > LN: arch/sim/src/board to > >> >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > >> >> > LN: include/arch/board to > >> >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > >> >> > LN: include/arch/chip to > >> >> > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > >> >> > LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/audioutils > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/boot > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/canutils > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/crypto > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/database > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/module > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/fsutils > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/games > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/graphics > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/industry > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/inertial > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/logging > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/lte > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/math > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/mlearning > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/netutils > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/sdr > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/system > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/tee > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/videoutils > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154 > >> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless > >> >> > mkkconfig in /home/ligd/platform/mainline/apps > >> >> > Loaded configuration '.config' > >> >> > Configuration saved to '.config' > >> >> > ligd@BAI:~/platform/mainline/nuttx$ > >> >> > *ligd@BAI:~/platform/mainline/nuttx$ make* > >> >> > Create version.h > >> >> > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to > >> >> > /home/ligd/platform/mainline/apps/platform/dummy > >> >> > Downloading: v1.46.0.zip > >> >> > ... > >> >> > patching file > >> >> > > >> >> > > >> >> > >> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > >> >> > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > >> >> > > >> >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > >> >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > >> >> > LD: nuttx > >> >> > Pac SIM with dynamic libs.. > >> >> > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1' > >> >> > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> > >> 'sim-pac/libs/libstdc++.so.6' > >> >> > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6' > >> >> > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6' > >> >> > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1' > >> >> > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2' > >> >> > SIM elf with dynamic libs archive in nuttx.tgz > >> >> > ligd@BAI:~/platform/mainline/nuttx$ > >> >> > ligd@BAI:~/platform/mainline/nuttx$ > >> >> > ligd@BAI:~/platform/mainline/nuttx$ > >> >> > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > >> >> > CHelloWorld: Constructor: mSecret=42 > >> >> > nsh: init: open failed: 2 > >> >> > > >> >> > NuttShell (NSH) NuttX-12.11.0 > >> >> > MOTD: username=admin password=Administrator > >> >> > nsh> > >> >> > nsh> help > >> >> > help usage: help [-v] [<cmd>] > >> >> > > >> >> > . df ifconfig mkfifo readlink umount > >> >> > [ dmesg ifdown mkrd rm unset > >> >> > ? echo ifup mount rmdir uptime > >> >> > alias env insmod mv rmmod usleep > >> >> > unalias exec kill nslookup set watch > >> >> > basename exit pkill pidof sleep xd > >> >> > break expr losetup pmconfig source wait > >> >> > cat false ln poweroff test > >> >> > cd fdinfo ls quit time > >> >> > cp free lsmod printf true > >> >> > cmp help mkdir ps truncate > >> >> > dirname hexdump mkfatfs pwd uname > >> >> > > >> >> > Builtin Apps: > >> >> > cmocka > >> >> > cmocka_driver_block > >> >> > cmocka_driver_gpio > >> >> > cmocka_driver_pm > >> >> > cmocka_driver_pm_runtime > >> >> > cmocka_driver_regulator > >> >> > > >> >> > Matteo Golin <[email protected]> 于2025年12月22日周一 10:15写道： > >> >> > > >> >> > > I think so. Best to leave OS test alone and slowly replace it with a > >> >> new > >> >> > > tool that has improvements. The only reason I can see to modify OS > >> test > >> >> > now > >> >> > > would just be to have clearer output about what passed/failed. > >> >> > > > >> >> > > I'll come up with an improved proof of concept for a new tool to > >> share > >> >> in > >> >> > > the next few days. > >> >> > > > >> >> > > Matteo > >> >> > > > >> >> > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <[email protected]> wrote: > >> >> > > > >> >> > > > Small steps with measurable results may be the best approach > >> here, to > >> >> > > > create some new solution next to existing ostest but not > >> replacing it > >> >> > > > until ready? :-) > >> >> > > > > >> >> > > > -- > >> >> > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > >> >> > > > > >> >> > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > >> [email protected] > >> >> > > >> >> > > > wrote: > >> >> > > > > > >> >> > > > > I agree, I think it would take a long time to have proper > >> coverage > >> >> of > >> >> > > > NuttX > >> >> > > > > as a whole. Perhaps even impossible, since Unity is more of a > >> unit > >> >> > > > testing > >> >> > > > > framework and thus there are definitely test cases for NuttX > >> that > >> >> > would > >> >> > > > be > >> >> > > > > outside of its realm of "applicability". There is still a need > >> for > >> >> a > >> >> > > > > variety of testing applications and bench-marking applications > >> for > >> >> > > those > >> >> > > > > scenarios. > >> >> > > > > > >> >> > > > > That makes sense, prerequisites would need to be considered. I > >> >> think > >> >> > > > > targeting the low-hanging fruit of unit-testable code would be > >> easy > >> >> > in > >> >> > > > that > >> >> > > > > case, since prerequisites are mostly encoded in the Kconfig > >> >> > definitions > >> >> > > > > existing or not for features to be tested. Any complex behaviour > >> >> > > > > (interrupts, multi-threading, etc.) might be out of the realm of > >> >> > what's > >> >> > > > > testable by Unity in the first place. We'll always have OS test > >> in > >> >> > that > >> >> > > > > case, but maybe then it's better to do what Alan suggested and > >> at > >> >> > least > >> >> > > > > make its output more clear in the meantime. I suppose RTOS > >> testing > >> >> > is a > >> >> > > > > whole field in itself. > >> >> > > > > > >> >> > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > >> [email protected]> > >> >> > > > wrote: > >> >> > > > > > >> >> > > > > > I think that the answer depends on a tradeoff between effort > >> and > >> >> > test > >> >> > > > > > coverage. I could imagine that a proper job could be > >> man-years > >> >> of > >> >> > > > effort. > >> >> > > > > > > >> >> > > > > > My only concern with a re-design is to be certain that tests > >> are > >> >> > > > > > arranged so that the prerequisites of later test are tested > >> >> first. > >> >> > > If > >> >> > > > > > tests are grouped by, for example, functional area, then > >> failures > >> >> > can > >> >> > > > be > >> >> > > > > > difficult to understand if there is a failure in an unverified > >> >> test > >> >> > > > > > prerequisite. > >> >> > > > > > > >> >> > > > > > Some of the "tests" are bogus and only exist to provide some > >> >> > minimum > >> >> > > > > > confidence that we have the functionality to run the test. > >> For > >> >> > > > > > example: Can I start tasks? Can I pass parameters? Can I > >> >> survive > >> >> > a > >> >> > > > > > context switch. Some of the tests were intended only to > >> support > >> >> > > debug. > >> >> > > > > > Some were created only to track down specific bugs. > >> >> > > > > > > >> >> > > > > > On 12/21/25 18:39, Matteo Golin wrote: > >> >> > > > > > > LTS being the Long-Term-Stable release of NuttX? > >> >> > > > > > > > >> >> > > > > > > I'm starting to think that it would be best to create new > >> suite > >> >> > of > >> >> > > > tests > >> >> > > > > > > using Unity, using what's in OS Test as a basis and then > >> >> > expanding > >> >> > > it > >> >> > > > > > > further. Greg has pointed out that OS test is not intended > >> to > >> >> be > >> >> > a > >> >> > > > > > complete > >> >> > > > > > > test, so maybe a more solid testing framework base in a new > >> >> form > >> >> > > > would be > >> >> > > > > > > in order. At that point it would also be good to document > >> and > >> >> > > educate > >> >> > > > > > users > >> >> > > > > > > about other test options, since OS test is a current > >> favourite > >> >> > for > >> >> > > > > > showing > >> >> > > > > > > PRs haven't broken anything. Maybe that's not the best > >> choice > >> >> > > anymore > >> >> > > > > > with > >> >> > > > > > > this information. > >> >> > > > > > > > >> >> > > > > > > I don't want to re-invent the wheel of what's already in > >> >> > > `testsuite`, > >> >> > > > > > but I > >> >> > > > > > > think Unity might end up being lighter-weight in the long > >> run? > >> >> > And > >> >> > > > it is > >> >> > > > > > > also not a Xiaomi owned/maintained framework so it may > >> receive > >> >> > > better > >> >> > > > > > > external support (i.e. more users rely on it?). I tried to > >> >> > compile > >> >> > > > the > >> >> > > > > > > sim:citest configuration to run the testsuite example and > >> see > >> >> > what > >> >> > > > it was > >> >> > > > > > > like, but I was met with a long list of C++ compilation > >> errors > >> >> > that > >> >> > > > I'd > >> >> > > > > > > rather not deal with. There also seems to be zero docs > >> about it > >> >> > or > >> >> > > > cmocka > >> >> > > > > > > in our NuttX docs at the moment. > >> >> > > > > > > > >> >> > > > > > > What do you think? Start a Unity test suite, add some > >> switches > >> >> to > >> >> > > > choose > >> >> > > > > > > which test cases get compiled into the binary, open for > >> >> > extensions > >> >> > > by > >> >> > > > > > users > >> >> > > > > > > who know more about different subsystems? This way we can > >> keep > >> >> OS > >> >> > > > test > >> >> > > > > > how > >> >> > > > > > > it is for a quick, low-resource sanity check of "board > >> bringup" > >> >> > > while > >> >> > > > > > > having a dedicated testing application for patches? Someone > >> >> > > changing > >> >> > > > file > >> >> > > > > > > system stuff can just compile to include file system test > >> cases > >> >> > and > >> >> > > > run > >> >> > > > > > > that suite only as proof that their PR works. The output is > >> >> > really > >> >> > > > easy > >> >> > > > > > to > >> >> > > > > > > glance at to determine pass/fail results. > >> >> > > > > > > > >> >> > > > > > > Anyways, here's my draft demonstrating how Unity works with > >> the > >> >> > FPU > >> >> > > > test > >> >> > > > > > > suite from OS test: > >> >> > https://github.com/apache/nuttx-apps/pull/3259 > >> >> > > > > > > > >> >> > > > > > > Matteo > >> >> > > > > > > > >> >> > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > >> [email protected]> > >> >> > > > wrote: > >> >> > > > > > > > >> >> > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > >> >> > [email protected] > >> >> > > > > >> >> > > > > > wrote: > >> >> > > > > > >>> (..) > >> >> > > > > > >>> Starting with bits and pieces of something like the LTS > >> might > >> >> > > make > >> >> > > > more > >> >> > > > > > >>> sense other than its licensing problems. Since it would > >> >> never > >> >> > be > >> >> > > > > > >>> included in a distributed binary, the licensing really > >> should > >> >> > not > >> >> > > > > > matter > >> >> > > > > > >>> much in practice. > >> >> > > > > > >> Unless its GPL and some company would like to have their > >> own > >> >> > > > internal > >> >> > > > > > >> bits and pieces.. luckily its MIT and the whole thing is > >> >> > > open-source > >> >> > > > > > >> anyways :-) > >> >> > > > > > >> > >> >> > > > > > >> Yes we had many discussions about the LTS and this could be > >> >> > great > >> >> > > > start! > >> >> > > > > > >> :-) > >> >> > > > > > >> > >> >> > > > > > >> -- > >> >> > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > >> >> > > > > > >> > >> >> > > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >>
