I think I will try to make a version of the sample I wrote which uses cmocka to compare a bit better and report back.
On Mon, Dec 22, 2025 at 1:28 PM Alan C. Assis <[email protected]> wrote: > Hi Pierre-Noel, > > The instructions are here: https://nuttx.apache.org/community/ > > We are glad for all the things you did for NuttX (EFM32 port, ESP8266, > etc.) > > Thank you very much! > > BR, > > Alan > > On Mon, Dec 22, 2025 at 3:16 PM Pierre-Noel Bouteville <[email protected] > > > wrote: > > > How to unsbcribe? > > Pierre-Noël Bouteville > > Envoyé de mon iPhone > > > > > Le 22 déc. 2025 à 19:13, raiden00pl <[email protected]> a écrit : > > > > > > Cmocka tests can run on any target. It doesn't matter if it's sim, > qemu > > or > > > real hardware. > > > You just need to prepare a defconfig and have the right amount of > FLASH. > > > > > > For small systems we can always try to reduce the size of the cmock, > e.g. > > > by disabling > > > unnecessary prints. I doubt that the logic of running tests in cmoka is > > > heavy, running > > > test cases are not a complicated operation, I think most of the > required > > > FLASH is wasted > > > on strings. > > > > > > NTFC supports serial port communication and sample test scenario based > on > > > nucleo-h743z > > > with cmocka tests enabled. The only thing missing for full test > > automation > > > (compile, flash, > > > test, result) is cmake support for testsuites. > > > > > > pon., 22 gru 2025 o 18:08 Matteo Golin <[email protected]> > > napisał(a): > > > > > >> Hello everyone, > > >> > > >> Thanks for all the feedback! I'm glad I discussed this here first. > I'll > > try > > >> to address everything that was brought up. > > >> > > >> Thanks Guiding for the input! I must just be missing some library on > my > > >> machine if you're able to compile it just fine. Maybe I'll track down > > that > > >> error and see if you can recognize what's wrong. > > >> > > >> I don't intend to replace OS test anymore. I think it serves its > > purpose as > > >> a minimal sanity check. Instead, I think we need something a little > more > > >> involved to test PRs since OS test can't catch many issues. It is > > currently > > >> the standard everyone uses for verifying their patches and it would be > > >> better to have more complete, alternate options. I do think that we > can > > all > > >> agree that OS test needs a revision of its console output so that it > is > > >> much easier to see pass/fail results, but that is just a matter of > print > > >> statements and can be done in a different effort. > > >> > > >> I think the overwhelming concern here is mixing two unit testing > > frameworks > > >> and no longer having one resource where we can go for tests. I share > > raiden > > >> and Guiding's concern about that, which is why I'm glad I opened for > > >> discussion here. On the one hand, Xiaomi has a pretty substantial > > >> collection of tests in the `testsuite` application which they are > using > > >> internally I believe (in their CI). These run well on virtual > machines, > > >> which I assume is why the only configurations with that test suite > > included > > >> is sim and rv-virt. I do not know nearly enough about cmocka to make > any > > >> claim that it *wouldn't* work on smaller hardware devices, but it also > > >> definitely looks to be less light-weight from the feature set in the > > >> documentation. On the other hand, the Xiaomi test suites are also not > a > > >> complete silver bullet since they seem not to be intended to run on > > >> hardware, and there are *a lot *of tests that need hardware > > verification. > > >> In PRs we often request more than just a simulator test for certain > > >> changes. For instance, sim isn't able to have `/dev/rand`, so that > > needs to > > >> be tested on a different target. > > >> > > >> Maybe someone more familiar with Xiaomi's testing practices can weigh > > in on > > >> whether cmocka has been run on hardware. Going purely based off of > their > > >> PRs, which almost exclusively cite tests on simulators, my guess is > that > > >> cmocka was not designed with embedded in mind when it was written. > > >> > > >> As for size: unit testing can get large with Unity or cmocka (as > raiden > > >> mentioned) as more tests are added. Hence my effort to enable/disable > > >> individual test suites from getting compiled. It might not be possible > > to > > >> ever get good testing ability for AVR just because the devices are so > > >> small. Whatever solution is used there is probably going to involve > > >> flashing the device multiple times with different tests. The size goal > > >> should just be to target the most NuttX devices possible. > > >> > > >> Then there are the pros and cons of each framework (again, I don't > know > > >> much about cmocka so take it with a grain of salt): > > >> > > >> Unity's cons: > > >> - Designed for each test-suite to be a standalone binary. This is > > clunky on > > >> NuttX, so `nest` is currently implemented as a mega-suite where test > > cases > > >> just follow a naming convention to make it clear what is being tested > > >> - setUp/tearDown can't really be leveraged because of the above > > >> - I've noticed some funky behaviour using TEST_IGNORE in certain > > scenarios > > >> but that's possibly my own configuration issue > > >> - Would be in conflict with existing cmocka tests > > >> > > >> Unity's pros: > > >> - Highly configurable, like NuttX > > >> - Light-weight and works for embedded devices > > >> - Significantly larger user base than cmocka > > >> > > >> Cmocka's cons: > > >> - Possibly (?) not light-weight > > >> - No documentation for existing test cases in NuttX > > >> - Small user base > > >> > > >> Cmocka's pros: > > >> - Seems to have a larger set of features, including setup/teardown > > >> functions per-test case > > >> - Existing tests in NuttX > > >> > > >> Irrespective of whether we go with Unity or cmocka, the collection of > > tests > > >> that we write will be intended to run on the actual hardware devices > > >> themselves for the purposes of testing patches and doing regression > > tests. > > >> As a result, the tests will be separate from the Xiaomi 'testsuite' > > >> application which is intended for CI testing. I think this > realistically > > >> nullifies choosing which framework to use on the basis of consistency, > > >> since the tests are not coexisting with the cmocka tests Xiaomi has > > >> written. Instead, the choice should be based on what best achieves the > > goal > > >> from a functionality perspective. Unity seems like a really great > choice > > >> for this in my opinion, but I am bothered by the fact that it's > designed > > >> with multiple test binaries in mind. > > >> > > >> Let me know what you think, > > >> Matteo > > >> > > >>> On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <[email protected]> > > wrote: > > >>> > > >>> Yes, I agree! We need data size from each solution for comparison, > this > > >> is > > >>> the first step. > > >>> > > >>> Maybe the conversion could be facilitated if we create equivalent > > >> functions > > >>> like those used in the apps/testing/testsuites: > > >>> > > >>> cmocka_run_group_tests -> unity_run_group_tests > > >>> cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown > > >>> etc... > > >>> > > >>> This way we avoid end up with two unit test framework, we already > have > > >> the > > >>> Makefile vs CMake precedent! :-D > > >>> > > >>> On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <[email protected]> > > >> wrote: > > >>> > > >>>> The discussion about small systems doesn't make much sense without > > >>>> data comparing resources of the same test cases for cmock and unity. > > >>>> > > >>>> But let's assume that Unity is better. Should we use two frameworks > > >>>> to implement unit tests for OS? Or choose one framework? If we > choose > > >>> only > > >>>> Unity, who > > >>>> will migrate the current cmocka tests to Unity? Remember that cmoka > > >> tests > > >>>> are used in CI, so all CI tools must be also migrated. > > >>>> Or should we use two different frameworks to test different things > and > > >>> make > > >>>> even > > >>>> more mess in apps/testing as it is now? > > >>>> > > >>>> No one is forcing anyone to do anything. This is an open-source > > >> project, > > >>>> everyone > > >>>> should do what they feel is right. But whether the change should be > > >>> merged > > >>>> is > > >>>> a different question. In this case, it's a very important question, > > >>> because > > >>>> unit testing > > >>>> isn't just about test cases. It's about the entire infrastructure > that > > >>>> runs, captures and > > >>>> handles test results later. Writing the test cases is the least of > the > > >>>> problems. > > >>>> The infrastructure for cmoka testing is already in place, the > > >>>> infrastructure for Unity > > >>>> testing isn't. > > >>>> > > >>>> pon., 22 gru 2025 o 14:16 Alan C. Assis <[email protected]> > > >> napisał(a): > > >>>> > > >>>>> Mateusz, > > >>>>> but some chips like AVR DA have big flash, but not so much RAM. > > >>>>> > > >>>>> It is better to have some possible way to test on a small system, > > >> using > > >>>>> cmocka we are removing this option from the table. > > >>>>> > > >>>>> I liked the test Matteo did with Unity and I think we should let > him > > >>>>> continue. Forcing him to migrate to cmocka just because there's a > > >> test > > >>>>> suite from Xiaomi doesn't seem prudent to me. > > >>>>> > > >>>>> This is the advantage of an open mind open-source project: "the > > >>>> possibility > > >>>>> to test new ideas", if someone had said to Michal Lenc he couldn't > > >>> create > > >>>>> NXBoot because where is already MCUBoot integrated on NuttX, we > > >> didn't > > >>>> have > > >>>>> the option to use NXBoot today. ;-) > > >>>>> > > >>>>> BR, > > >>>>> > > >>>>> Alan > > >>>>> > > >>>>> On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <[email protected]> > > >>>> wrote: > > >>>>> > > >>>>>> Unit testing by itself isn't suitable for small systems. Ostest is > > >>>>>> something > > >>>>>> different (system level testing), so running it on small systems > > >>> makes > > >>>>>> sense. > > >>>>>> Running unit tests on small systems is pointless because you won't > > >>> fit > > >>>> in > > >>>>>> FLASH > > >>>>>> either way. You can run the same on large targets (preferably on > > >>> host: > > >>>>> sim, > > >>>>>> qemu) > > >>>>>> and achieve the same result. > > >>>>>> > > >>>>>> pon., 22 gru 2025 o 13:35 Alan C. Assis <[email protected]> > > >>>> napisał(a): > > >>>>>> > > >>>>>>> Is it too complicated to rewrite the cmocka tests in Unity? > > >>>>>>> > > >>>>>>> If cmoka is not feasible for small microcontrollers, like AVRs > > >> and > > >>>>>> others, > > >>>>>>> our test coverage will be faulty. > > >>>>>>> > > >>>>>>> BR, > > >>>>>>> > > >>>>>>> Alan > > >>>>>>> > > >>>>>>> On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <[email protected] > > >>> > > >>>>> wrote: > > >>>>>>> > > >>>>>>>> I'm a fan of Unity and have been using it in my projects for > > >>> years, > > >>>>>> but I > > >>>>>>>> think > > >>>>>>>> we should stick with one testing framework. Currently, Cmocka > > >> is > > >>>> used > > >>>>>> in > > >>>>>>>> nuttx-apps, and we should stick with it. The only question is > > >> how > > >>>>> well > > >>>>>>>> Cmocka > > >>>>>>>> is suitable for running on small systems? Will ostest still be > > >>> able > > >>>>> to > > >>>>>>> run > > >>>>>>>> on small MCUs? > > >>>>>>>> With Unity this would be quite likely, so maybe an exception > > >> for > > >>>>> Unity > > >>>>>>> and > > >>>>>>>> ostest would > > >>>>>>>> be acceptable, but using it for other test cases will only > > >> cause > > >>>>> chaos. > > >>>>>>>> > > >>>>>>>> pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> > > >>>> napisał(a): > > >>>>>>>> > > >>>>>>>>> Hi Matteo, > > >>>>>>>>> > > >>>>>>>>> Good idea ! > > >>>>>>>>> > > >>>>>>>>> But it is best to stick with a single testing framework. > > >>>> Currently, > > >>>>>>> many > > >>>>>>>>> test cases are using cmocka, so I suggest reusing cmocka for > > >>>>>>> consistency. > > >>>>>>>>> > > >>>>>>>>> ostest is a minimal basic test suite, typically used for CT > > >>>>>> (continuous > > >>>>>>>>> testing) of each patch. > > >>>>>>>>> If we move ostest to cmocka, it would make ostest dependent > > >> on > > >>> a > > >>>>>>> specific > > >>>>>>>>> framework before it can be run. > > >>>>>>>>> And It may increase the code size, which is not very friendly > > >>> for > > >>>>>>>>> configurations with small flash memory. > > >>>>>>>>> > > >>>>>>>>> As Greg mentioned: the answer depends on a tradeoff between > > >>>> effort > > >>>>>> and > > >>>>>>>> test > > >>>>>>>>> coverage. > > >>>>>>>>> > > >>>>>>>>> We need to clarify whether we define ostest as a lightweight, > > >>>>>>>> ready-to-run > > >>>>>>>>> test set for quick verification, or as part of the full test > > >>>> suite. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> The follow is my compiling for cmocka: > > >>>>>>>>> > > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh > > >>>>> sim:citest > > >>>>>>>> -j8* > > >>>>>>>>> Copy files > > >>>>>>>>> Select CONFIG_HOST_LINUX=y > > >>>>>>>>> Refreshing... > > >>>>>>>>> CP: arch/dummy/Kconfig to > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > > >>>>>>>>> CP: boards/dummy/Kconfig to > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > > >>>>>>>>> LN: include/arch to arch/sim/include > > >>>>>>>>> LN: drivers/platform to > > >>>>>>> /home/ligd/platform/mainline/nuttx/drivers/dummy > > >>>>>>>>> LN: arch/sim/src/chip to > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > > >>>>>>>>> LN: arch/sim/src/board to > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > > >>>>>>>>> LN: include/arch/board to > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > > >>>>>>>>> LN: include/arch/chip to > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > > >>>>>>>>> LN: platform/board to > > >>>>>> /home/ligd/platform/mainline/apps/platform/dummy > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/audioutils > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/boot > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/canutils > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/crypto > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/database > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples/rust > > >>>>>>>>> mkkconfig in > > >> /home/ligd/platform/mainline/apps/examples/sotest > > >>>>>>>>> mkkconfig in > > >> /home/ligd/platform/mainline/apps/examples/module > > >>>>>>>>> mkkconfig in > > >> /home/ligd/platform/mainline/apps/examples/mcuboot > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/fsutils > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/games > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/graphics > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/industry > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/inertial > > >>>>>>>>> mkkconfig in > > >>>>>> /home/ligd/platform/mainline/apps/interpreters/luamodules > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/interpreters > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/logging > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/lte > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/math > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/mlearning > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/netutils > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/sdr > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/system > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/tee > > >>>>>>>>> mkkconfig in > > >> /home/ligd/platform/mainline/apps/testing/drivers > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/sched > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/libc > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/arch > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/videoutils > > >>>>>>>>> mkkconfig in > > >>> /home/ligd/platform/mainline/apps/wireless/bluetooth > > >>>>>>>>> mkkconfig in > > >>>> /home/ligd/platform/mainline/apps/wireless/ieee802154 > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/wireless > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps > > >>>>>>>>> Loaded configuration '.config' > > >>>>>>>>> Configuration saved to '.config' > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ make* > > >>>>>>>>> Create version.h > > >>>>>>>>> Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to > > >>>>>>>>> /home/ligd/platform/mainline/apps/platform/dummy > > >>>>>>>>> Downloading: v1.46.0.zip > > >>>>>>>>> ... > > >>>>>>>>> patching file > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > > >>>>>>>>> Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > > >>>>>>>>> > > >>>>>>>>> CP: > > >> /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > > >>>>>>>>> CP: > > >>> /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > > >>>>>>>>> LD: nuttx > > >>>>>>>>> Pac SIM with dynamic libs.. > > >>>>>>>>> '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1' > > >>>>>>>>> '/lib/x86_64-linux-gnu/libstdc++.so.6' -> > > >>>>>> 'sim-pac/libs/libstdc++.so.6' > > >>>>>>>>> '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6' > > >>>>>>>>> '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6' > > >>>>>>>>> '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> > > >>>>> 'sim-pac/libs/libgcc_s.so.1' > > >>>>>>>>> '/lib64/ld-linux-x86-64.so.2' -> > > >> 'sim-pac/ld-linux-x86-64.so.2' > > >>>>>>>>> SIM elf with dynamic libs archive in nuttx.tgz > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > > >>>>>>>>> CHelloWorld: Constructor: mSecret=42 > > >>>>>>>>> nsh: init: open failed: 2 > > >>>>>>>>> > > >>>>>>>>> NuttShell (NSH) NuttX-12.11.0 > > >>>>>>>>> MOTD: username=admin password=Administrator > > >>>>>>>>> nsh> > > >>>>>>>>> nsh> help > > >>>>>>>>> help usage: help [-v] [<cmd>] > > >>>>>>>>> > > >>>>>>>>> . df ifconfig mkfifo readlink > > >>>>>> umount > > >>>>>>>>> [ dmesg ifdown mkrd rm > > >>>>> unset > > >>>>>>>>> ? echo ifup mount rmdir > > >>>>>> uptime > > >>>>>>>>> alias env insmod mv rmmod > > >>>>>> usleep > > >>>>>>>>> unalias exec kill nslookup set > > >>>>> watch > > >>>>>>>>> basename exit pkill pidof sleep > > >>>> xd > > >>>>>>>>> break expr losetup pmconfig source > > >>>>> wait > > >>>>>>>>> cat false ln poweroff test > > >>>>>>>>> cd fdinfo ls quit time > > >>>>>>>>> cp free lsmod printf true > > >>>>>>>>> cmp help mkdir ps truncate > > >>>>>>>>> dirname hexdump mkfatfs pwd uname > > >>>>>>>>> > > >>>>>>>>> Builtin Apps: > > >>>>>>>>> cmocka > > >>>>>>>>> cmocka_driver_block > > >>>>>>>>> cmocka_driver_gpio > > >>>>>>>>> cmocka_driver_pm > > >>>>>>>>> cmocka_driver_pm_runtime > > >>>>>>>>> cmocka_driver_regulator > > >>>>>>>>> > > >>>>>>>>> Matteo Golin <[email protected]> 于2025年12月22日周一 > > >> 10:15写道： > > >>>>>>>>> > > >>>>>>>>>> I think so. Best to leave OS test alone and slowly replace > > >> it > > >>>>> with > > >>>>>> a > > >>>>>>>> new > > >>>>>>>>>> tool that has improvements. The only reason I can see to > > >>> modify > > >>>>> OS > > >>>>>>> test > > >>>>>>>>> now > > >>>>>>>>>> would just be to have clearer output about what > > >>> passed/failed. > > >>>>>>>>>> > > >>>>>>>>>> I'll come up with an improved proof of concept for a new > > >> tool > > >>>> to > > >>>>>>> share > > >>>>>>>> in > > >>>>>>>>>> the next few days. > > >>>>>>>>>> > > >>>>>>>>>> Matteo > > >>>>>>>>>> > > >>>>>>>>>> On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO < > > >> [email protected]> > > >>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Small steps with measurable results may be the best > > >>> approach > > >>>>>> here, > > >>>>>>> to > > >>>>>>>>>>> create some new solution next to existing ostest but not > > >>>>>> replacing > > >>>>>>> it > > >>>>>>>>>>> until ready? :-) > > >>>>>>>>>>> > > >>>>>>>>>>> -- > > >>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > >>>>>>>>>>> > > >>>>>>>>>>> On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > > >>>>>>> [email protected] > > >>>>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>> I agree, I think it would take a long time to have > > >> proper > > >>>>>>> coverage > > >>>>>>>> of > > >>>>>>>>>>> NuttX > > >>>>>>>>>>>> as a whole. Perhaps even impossible, since Unity is > > >> more > > >>>> of a > > >>>>>>> unit > > >>>>>>>>>>> testing > > >>>>>>>>>>>> framework and thus there are definitely test cases for > > >>>> NuttX > > >>>>>> that > > >>>>>>>>> would > > >>>>>>>>>>> be > > >>>>>>>>>>>> outside of its realm of "applicability". There is > > >> still a > > >>>>> need > > >>>>>>> for > > >>>>>>>> a > > >>>>>>>>>>>> variety of testing applications and bench-marking > > >>>>> applications > > >>>>>>> for > > >>>>>>>>>> those > > >>>>>>>>>>>> scenarios. > > >>>>>>>>>>>> > > >>>>>>>>>>>> That makes sense, prerequisites would need to be > > >>>> considered. > > >>>>> I > > >>>>>>>> think > > >>>>>>>>>>>> targeting the low-hanging fruit of unit-testable code > > >>> would > > >>>>> be > > >>>>>>> easy > > >>>>>>>>> in > > >>>>>>>>>>> that > > >>>>>>>>>>>> case, since prerequisites are mostly encoded in the > > >>> Kconfig > > >>>>>>>>> definitions > > >>>>>>>>>>>> existing or not for features to be tested. Any complex > > >>>>>> behaviour > > >>>>>>>>>>>> (interrupts, multi-threading, etc.) might be out of the > > >>>> realm > > >>>>>> of > > >>>>>>>>> what's > > >>>>>>>>>>>> testable by Unity in the first place. We'll always have > > >>> OS > > >>>>> test > > >>>>>>> in > > >>>>>>>>> that > > >>>>>>>>>>>> case, but maybe then it's better to do what Alan > > >>> suggested > > >>>>> and > > >>>>>> at > > >>>>>>>>> least > > >>>>>>>>>>>> make its output more clear in the meantime. I suppose > > >>> RTOS > > >>>>>>> testing > > >>>>>>>>> is a > > >>>>>>>>>>>> whole field in itself. > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > > >>>>>>> [email protected]> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> I think that the answer depends on a tradeoff between > > >>>>> effort > > >>>>>>> and > > >>>>>>>>> test > > >>>>>>>>>>>>> coverage. I could imagine that a proper job could be > > >>>>>> man-years > > >>>>>>>> of > > >>>>>>>>>>> effort. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> My only concern with a re-design is to be certain > > >> that > > >>>>> tests > > >>>>>>> are > > >>>>>>>>>>>>> arranged so that the prerequisites of later test are > > >>>> tested > > >>>>>>>> first. > > >>>>>>>>>> If > > >>>>>>>>>>>>> tests are grouped by, for example, functional area, > > >>> then > > >>>>>>> failures > > >>>>>>>>> can > > >>>>>>>>>>> be > > >>>>>>>>>>>>> difficult to understand if there is a failure in an > > >>>>>> unverified > > >>>>>>>> test > > >>>>>>>>>>>>> prerequisite. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Some of the "tests" are bogus and only exist to > > >> provide > > >>>>> some > > >>>>>>>>> minimum > > >>>>>>>>>>>>> confidence that we have the functionality to run the > > >>>> test. > > >>>>>> For > > >>>>>>>>>>>>> example: Can I start tasks? Can I pass parameters? > > >>> Can > > >>>> I > > >>>>>>>> survive > > >>>>>>>>> a > > >>>>>>>>>>>>> context switch. Some of the tests were intended only > > >>> to > > >>>>>>> support > > >>>>>>>>>> debug. > > >>>>>>>>>>>>> Some were created only to track down specific bugs. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On 12/21/25 18:39, Matteo Golin wrote: > > >>>>>>>>>>>>>> LTS being the Long-Term-Stable release of NuttX? > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> I'm starting to think that it would be best to > > >> create > > >>>> new > > >>>>>>> suite > > >>>>>>>>> of > > >>>>>>>>>>> tests > > >>>>>>>>>>>>>> using Unity, using what's in OS Test as a basis and > > >>>> then > > >>>>>>>>> expanding > > >>>>>>>>>> it > > >>>>>>>>>>>>>> further. Greg has pointed out that OS test is not > > >>>>> intended > > >>>>>> to > > >>>>>>>> be > > >>>>>>>>> a > > >>>>>>>>>>>>> complete > > >>>>>>>>>>>>>> test, so maybe a more solid testing framework base > > >>> in a > > >>>>> new > > >>>>>>>> form > > >>>>>>>>>>> would be > > >>>>>>>>>>>>>> in order. At that point it would also be good to > > >>>> document > > >>>>>> and > > >>>>>>>>>> educate > > >>>>>>>>>>>>> users > > >>>>>>>>>>>>>> about other test options, since OS test is a > > >> current > > >>>>>>> favourite > > >>>>>>>>> for > > >>>>>>>>>>>>> showing > > >>>>>>>>>>>>>> PRs haven't broken anything. Maybe that's not the > > >>> best > > >>>>>> choice > > >>>>>>>>>> anymore > > >>>>>>>>>>>>> with > > >>>>>>>>>>>>>> this information. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> I don't want to re-invent the wheel of what's > > >> already > > >>>> in > > >>>>>>>>>> `testsuite`, > > >>>>>>>>>>>>> but I > > >>>>>>>>>>>>>> think Unity might end up being lighter-weight in > > >> the > > >>>> long > > >>>>>>> run? > > >>>>>>>>> And > > >>>>>>>>>>> it is > > >>>>>>>>>>>>>> also not a Xiaomi owned/maintained framework so it > > >>> may > > >>>>>>> receive > > >>>>>>>>>> better > > >>>>>>>>>>>>>> external support (i.e. more users rely on it?). I > > >>> tried > > >>>>> to > > >>>>>>>>> compile > > >>>>>>>>>>> the > > >>>>>>>>>>>>>> sim:citest configuration to run the testsuite > > >> example > > >>>> and > > >>>>>> see > > >>>>>>>>> what > > >>>>>>>>>>> it was > > >>>>>>>>>>>>>> like, but I was met with a long list of C++ > > >>> compilation > > >>>>>>> errors > > >>>>>>>>> that > > >>>>>>>>>>> I'd > > >>>>>>>>>>>>>> rather not deal with. There also seems to be zero > > >>> docs > > >>>>>> about > > >>>>>>> it > > >>>>>>>>> or > > >>>>>>>>>>> cmocka > > >>>>>>>>>>>>>> in our NuttX docs at the moment. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> What do you think? Start a Unity test suite, add > > >> some > > >>>>>>> switches > > >>>>>>>> to > > >>>>>>>>>>> choose > > >>>>>>>>>>>>>> which test cases get compiled into the binary, open > > >>> for > > >>>>>>>>> extensions > > >>>>>>>>>> by > > >>>>>>>>>>>>> users > > >>>>>>>>>>>>>> who know more about different subsystems? This way > > >> we > > >>>> can > > >>>>>>> keep > > >>>>>>>> OS > > >>>>>>>>>>> test > > >>>>>>>>>>>>> how > > >>>>>>>>>>>>>> it is for a quick, low-resource sanity check of > > >>> "board > > >>>>>>> bringup" > > >>>>>>>>>> while > > >>>>>>>>>>>>>> having a dedicated testing application for patches? > > >>>>> Someone > > >>>>>>>>>> changing > > >>>>>>>>>>> file > > >>>>>>>>>>>>>> system stuff can just compile to include file > > >> system > > >>>> test > > >>>>>>> cases > > >>>>>>>>> and > > >>>>>>>>>>> run > > >>>>>>>>>>>>>> that suite only as proof that their PR works. The > > >>>> output > > >>>>> is > > >>>>>>>>> really > > >>>>>>>>>>> easy > > >>>>>>>>>>>>> to > > >>>>>>>>>>>>>> glance at to determine pass/fail results. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Anyways, here's my draft demonstrating how Unity > > >>> works > > >>>>> with > > >>>>>>> the > > >>>>>>>>> FPU > > >>>>>>>>>>> test > > >>>>>>>>>>>>>> suite from OS test: > > >>>>>>>>> https://github.com/apache/nuttx-apps/pull/3259 > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Matteo > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > > >>>>>>> [email protected]> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > > >>>>>>>>> [email protected] > > >>>>>>>>>>> > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>> (..) > > >>>>>>>>>>>>>>>> Starting with bits and pieces of something like > > >> the > > >>>> LTS > > >>>>>>> might > > >>>>>>>>>> make > > >>>>>>>>>>> more > > >>>>>>>>>>>>>>>> sense other than its licensing problems. Since > > >> it > > >>>>> would > > >>>>>>>> never > > >>>>>>>>> be > > >>>>>>>>>>>>>>>> included in a distributed binary, the licensing > > >>>> really > > >>>>>>> should > > >>>>>>>>> not > > >>>>>>>>>>>>> matter > > >>>>>>>>>>>>>>>> much in practice. > > >>>>>>>>>>>>>>> Unless its GPL and some company would like to have > > >>>> their > > >>>>>> own > > >>>>>>>>>>> internal > > >>>>>>>>>>>>>>> bits and pieces.. luckily its MIT and the whole > > >>> thing > > >>>> is > > >>>>>>>>>> open-source > > >>>>>>>>>>>>>>> anyways :-) > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Yes we had many discussions about the LTS and this > > >>>> could > > >>>>>> be > > >>>>>>>>> great > > >>>>>>>>>>> start! > > >>>>>>>>>>>>>>> :-) > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > >
