Yes, that is a good idea! On Mon, Dec 22, 2025 at 5:56 PM Matteo Golin <[email protected]> wrote:
> I think I will try to make a version of the sample I wrote which uses > cmocka to compare a bit better and report back. > > On Mon, Dec 22, 2025 at 1:28 PM Alan C. Assis <[email protected]> wrote: > > > Hi Pierre-Noel, > > > > The instructions are here: https://nuttx.apache.org/community/ > > > > We are glad for all the things you did for NuttX (EFM32 port, ESP8266, > > etc.) > > > > Thank you very much! > > > > BR, > > > > Alan > > > > On Mon, Dec 22, 2025 at 3:16 PM Pierre-Noel Bouteville < > [email protected] > > > > > wrote: > > > > > How to unsbcribe? > > > Pierre-Noël Bouteville > > > Envoyé de mon iPhone > > > > > > > Le 22 déc. 2025 à 19:13, raiden00pl <[email protected]> a écrit : > > > > > > > > Cmocka tests can run on any target. It doesn't matter if it's sim, > > qemu > > > or > > > > real hardware. > > > > You just need to prepare a defconfig and have the right amount of > > FLASH. > > > > > > > > For small systems we can always try to reduce the size of the cmock, > > e.g. > > > > by disabling > > > > unnecessary prints. I doubt that the logic of running tests in cmoka > is > > > > heavy, running > > > > test cases are not a complicated operation, I think most of the > > required > > > > FLASH is wasted > > > > on strings. > > > > > > > > NTFC supports serial port communication and sample test scenario > based > > on > > > > nucleo-h743z > > > > with cmocka tests enabled. The only thing missing for full test > > > automation > > > > (compile, flash, > > > > test, result) is cmake support for testsuites. > > > > > > > > pon., 22 gru 2025 o 18:08 Matteo Golin <[email protected]> > > > napisał(a): > > > > > > > >> Hello everyone, > > > >> > > > >> Thanks for all the feedback! I'm glad I discussed this here first. > > I'll > > > try > > > >> to address everything that was brought up. > > > >> > > > >> Thanks Guiding for the input! I must just be missing some library on > > my > > > >> machine if you're able to compile it just fine. Maybe I'll track > down > > > that > > > >> error and see if you can recognize what's wrong. > > > >> > > > >> I don't intend to replace OS test anymore. I think it serves its > > > purpose as > > > >> a minimal sanity check. Instead, I think we need something a little > > more > > > >> involved to test PRs since OS test can't catch many issues. It is > > > currently > > > >> the standard everyone uses for verifying their patches and it would > be > > > >> better to have more complete, alternate options. I do think that we > > can > > > all > > > >> agree that OS test needs a revision of its console output so that it > > is > > > >> much easier to see pass/fail results, but that is just a matter of > > print > > > >> statements and can be done in a different effort. > > > >> > > > >> I think the overwhelming concern here is mixing two unit testing > > > frameworks > > > >> and no longer having one resource where we can go for tests. I share > > > raiden > > > >> and Guiding's concern about that, which is why I'm glad I opened for > > > >> discussion here. On the one hand, Xiaomi has a pretty substantial > > > >> collection of tests in the `testsuite` application which they are > > using > > > >> internally I believe (in their CI). These run well on virtual > > machines, > > > >> which I assume is why the only configurations with that test suite > > > included > > > >> is sim and rv-virt. I do not know nearly enough about cmocka to make > > any > > > >> claim that it *wouldn't* work on smaller hardware devices, but it > also > > > >> definitely looks to be less light-weight from the feature set in the > > > >> documentation. On the other hand, the Xiaomi test suites are also > not > > a > > > >> complete silver bullet since they seem not to be intended to run on > > > >> hardware, and there are *a lot *of tests that need hardware > > > verification. > > > >> In PRs we often request more than just a simulator test for certain > > > >> changes. For instance, sim isn't able to have `/dev/rand`, so that > > > needs to > > > >> be tested on a different target. > > > >> > > > >> Maybe someone more familiar with Xiaomi's testing practices can > weigh > > > in on > > > >> whether cmocka has been run on hardware. Going purely based off of > > their > > > >> PRs, which almost exclusively cite tests on simulators, my guess is > > that > > > >> cmocka was not designed with embedded in mind when it was written. > > > >> > > > >> As for size: unit testing can get large with Unity or cmocka (as > > raiden > > > >> mentioned) as more tests are added. Hence my effort to > enable/disable > > > >> individual test suites from getting compiled. It might not be > possible > > > to > > > >> ever get good testing ability for AVR just because the devices are > so > > > >> small. Whatever solution is used there is probably going to involve > > > >> flashing the device multiple times with different tests. The size > goal > > > >> should just be to target the most NuttX devices possible. > > > >> > > > >> Then there are the pros and cons of each framework (again, I don't > > know > > > >> much about cmocka so take it with a grain of salt): > > > >> > > > >> Unity's cons: > > > >> - Designed for each test-suite to be a standalone binary. This is > > > clunky on > > > >> NuttX, so `nest` is currently implemented as a mega-suite where test > > > cases > > > >> just follow a naming convention to make it clear what is being > tested > > > >> - setUp/tearDown can't really be leveraged because of the above > > > >> - I've noticed some funky behaviour using TEST_IGNORE in certain > > > scenarios > > > >> but that's possibly my own configuration issue > > > >> - Would be in conflict with existing cmocka tests > > > >> > > > >> Unity's pros: > > > >> - Highly configurable, like NuttX > > > >> - Light-weight and works for embedded devices > > > >> - Significantly larger user base than cmocka > > > >> > > > >> Cmocka's cons: > > > >> - Possibly (?) not light-weight > > > >> - No documentation for existing test cases in NuttX > > > >> - Small user base > > > >> > > > >> Cmocka's pros: > > > >> - Seems to have a larger set of features, including setup/teardown > > > >> functions per-test case > > > >> - Existing tests in NuttX > > > >> > > > >> Irrespective of whether we go with Unity or cmocka, the collection > of > > > tests > > > >> that we write will be intended to run on the actual hardware devices > > > >> themselves for the purposes of testing patches and doing regression > > > tests. > > > >> As a result, the tests will be separate from the Xiaomi 'testsuite' > > > >> application which is intended for CI testing. I think this > > realistically > > > >> nullifies choosing which framework to use on the basis of > consistency, > > > >> since the tests are not coexisting with the cmocka tests Xiaomi has > > > >> written. Instead, the choice should be based on what best achieves > the > > > goal > > > >> from a functionality perspective. Unity seems like a really great > > choice > > > >> for this in my opinion, but I am bothered by the fact that it's > > designed > > > >> with multiple test binaries in mind. > > > >> > > > >> Let me know what you think, > > > >> Matteo > > > >> > > > >>> On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <[email protected]> > > > wrote: > > > >>> > > > >>> Yes, I agree! We need data size from each solution for comparison, > > this > > > >> is > > > >>> the first step. > > > >>> > > > >>> Maybe the conversion could be facilitated if we create equivalent > > > >> functions > > > >>> like those used in the apps/testing/testsuites: > > > >>> > > > >>> cmocka_run_group_tests -> unity_run_group_tests > > > >>> cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown > > > >>> etc... > > > >>> > > > >>> This way we avoid end up with two unit test framework, we already > > have > > > >> the > > > >>> Makefile vs CMake precedent! :-D > > > >>> > > > >>> On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <[email protected]> > > > >> wrote: > > > >>> > > > >>>> The discussion about small systems doesn't make much sense without > > > >>>> data comparing resources of the same test cases for cmock and > unity. > > > >>>> > > > >>>> But let's assume that Unity is better. Should we use two > frameworks > > > >>>> to implement unit tests for OS? Or choose one framework? If we > > choose > > > >>> only > > > >>>> Unity, who > > > >>>> will migrate the current cmocka tests to Unity? Remember that > cmoka > > > >> tests > > > >>>> are used in CI, so all CI tools must be also migrated. > > > >>>> Or should we use two different frameworks to test different things > > and > > > >>> make > > > >>>> even > > > >>>> more mess in apps/testing as it is now? > > > >>>> > > > >>>> No one is forcing anyone to do anything. This is an open-source > > > >> project, > > > >>>> everyone > > > >>>> should do what they feel is right. But whether the change should > be > > > >>> merged > > > >>>> is > > > >>>> a different question. In this case, it's a very important > question, > > > >>> because > > > >>>> unit testing > > > >>>> isn't just about test cases. It's about the entire infrastructure > > that > > > >>>> runs, captures and > > > >>>> handles test results later. Writing the test cases is the least of > > the > > > >>>> problems. > > > >>>> The infrastructure for cmoka testing is already in place, the > > > >>>> infrastructure for Unity > > > >>>> testing isn't. > > > >>>> > > > >>>> pon., 22 gru 2025 o 14:16 Alan C. Assis <[email protected]> > > > >> napisał(a): > > > >>>> > > > >>>>> Mateusz, > > > >>>>> but some chips like AVR DA have big flash, but not so much RAM. > > > >>>>> > > > >>>>> It is better to have some possible way to test on a small system, > > > >> using > > > >>>>> cmocka we are removing this option from the table. > > > >>>>> > > > >>>>> I liked the test Matteo did with Unity and I think we should let > > him > > > >>>>> continue. Forcing him to migrate to cmocka just because there's a > > > >> test > > > >>>>> suite from Xiaomi doesn't seem prudent to me. > > > >>>>> > > > >>>>> This is the advantage of an open mind open-source project: "the > > > >>>> possibility > > > >>>>> to test new ideas", if someone had said to Michal Lenc he > couldn't > > > >>> create > > > >>>>> NXBoot because where is already MCUBoot integrated on NuttX, we > > > >> didn't > > > >>>> have > > > >>>>> the option to use NXBoot today. ;-) > > > >>>>> > > > >>>>> BR, > > > >>>>> > > > >>>>> Alan > > > >>>>> > > > >>>>> On Mon, Dec 22, 2025 at 10:02 AM raiden00pl < > [email protected]> > > > >>>> wrote: > > > >>>>> > > > >>>>>> Unit testing by itself isn't suitable for small systems. Ostest > is > > > >>>>>> something > > > >>>>>> different (system level testing), so running it on small systems > > > >>> makes > > > >>>>>> sense. > > > >>>>>> Running unit tests on small systems is pointless because you > won't > > > >>> fit > > > >>>> in > > > >>>>>> FLASH > > > >>>>>> either way. You can run the same on large targets (preferably on > > > >>> host: > > > >>>>> sim, > > > >>>>>> qemu) > > > >>>>>> and achieve the same result. > > > >>>>>> > > > >>>>>> pon., 22 gru 2025 o 13:35 Alan C. Assis <[email protected]> > > > >>>> napisał(a): > > > >>>>>> > > > >>>>>>> Is it too complicated to rewrite the cmocka tests in Unity? > > > >>>>>>> > > > >>>>>>> If cmoka is not feasible for small microcontrollers, like AVRs > > > >> and > > > >>>>>> others, > > > >>>>>>> our test coverage will be faulty. > > > >>>>>>> > > > >>>>>>> BR, > > > >>>>>>> > > > >>>>>>> Alan > > > >>>>>>> > > > >>>>>>> On Mon, Dec 22, 2025 at 6:14 AM raiden00pl < > [email protected] > > > >>> > > > >>>>> wrote: > > > >>>>>>> > > > >>>>>>>> I'm a fan of Unity and have been using it in my projects for > > > >>> years, > > > >>>>>> but I > > > >>>>>>>> think > > > >>>>>>>> we should stick with one testing framework. Currently, Cmocka > > > >> is > > > >>>> used > > > >>>>>> in > > > >>>>>>>> nuttx-apps, and we should stick with it. The only question is > > > >> how > > > >>>>> well > > > >>>>>>>> Cmocka > > > >>>>>>>> is suitable for running on small systems? Will ostest still be > > > >>> able > > > >>>>> to > > > >>>>>>> run > > > >>>>>>>> on small MCUs? > > > >>>>>>>> With Unity this would be quite likely, so maybe an exception > > > >> for > > > >>>>> Unity > > > >>>>>>> and > > > >>>>>>>> ostest would > > > >>>>>>>> be acceptable, but using it for other test cases will only > > > >> cause > > > >>>>> chaos. > > > >>>>>>>> > > > >>>>>>>> pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> > > > >>>> napisał(a): > > > >>>>>>>> > > > >>>>>>>>> Hi Matteo, > > > >>>>>>>>> > > > >>>>>>>>> Good idea ! > > > >>>>>>>>> > > > >>>>>>>>> But it is best to stick with a single testing framework. > > > >>>> Currently, > > > >>>>>>> many > > > >>>>>>>>> test cases are using cmocka, so I suggest reusing cmocka for > > > >>>>>>> consistency. > > > >>>>>>>>> > > > >>>>>>>>> ostest is a minimal basic test suite, typically used for CT > > > >>>>>> (continuous > > > >>>>>>>>> testing) of each patch. > > > >>>>>>>>> If we move ostest to cmocka, it would make ostest dependent > > > >> on > > > >>> a > > > >>>>>>> specific > > > >>>>>>>>> framework before it can be run. > > > >>>>>>>>> And It may increase the code size, which is not very friendly > > > >>> for > > > >>>>>>>>> configurations with small flash memory. > > > >>>>>>>>> > > > >>>>>>>>> As Greg mentioned: the answer depends on a tradeoff between > > > >>>> effort > > > >>>>>> and > > > >>>>>>>> test > > > >>>>>>>>> coverage. > > > >>>>>>>>> > > > >>>>>>>>> We need to clarify whether we define ostest as a lightweight, > > > >>>>>>>> ready-to-run > > > >>>>>>>>> test set for quick verification, or as part of the full test > > > >>>> suite. > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> The follow is my compiling for cmocka: > > > >>>>>>>>> > > > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh > > > >>>>> sim:citest > > > >>>>>>>> -j8* > > > >>>>>>>>> Copy files > > > >>>>>>>>> Select CONFIG_HOST_LINUX=y > > > >>>>>>>>> Refreshing... > > > >>>>>>>>> CP: arch/dummy/Kconfig to > > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > > > >>>>>>>>> CP: boards/dummy/Kconfig to > > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > > > >>>>>>>>> LN: include/arch to arch/sim/include > > > >>>>>>>>> LN: drivers/platform to > > > >>>>>>> /home/ligd/platform/mainline/nuttx/drivers/dummy > > > >>>>>>>>> LN: arch/sim/src/chip to > > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > > > >>>>>>>>> LN: arch/sim/src/board to > > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > > > >>>>>>>>> LN: include/arch/board to > > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > > > >>>>>>>>> LN: include/arch/chip to > > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > > > >>>>>>>>> LN: platform/board to > > > >>>>>> /home/ligd/platform/mainline/apps/platform/dummy > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/audioutils > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/boot > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/canutils > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/crypto > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/database > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples/rust > > > >>>>>>>>> mkkconfig in > > > >> /home/ligd/platform/mainline/apps/examples/sotest > > > >>>>>>>>> mkkconfig in > > > >> /home/ligd/platform/mainline/apps/examples/module > > > >>>>>>>>> mkkconfig in > > > >> /home/ligd/platform/mainline/apps/examples/mcuboot > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/fsutils > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/games > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/graphics > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/industry > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/inertial > > > >>>>>>>>> mkkconfig in > > > >>>>>> /home/ligd/platform/mainline/apps/interpreters/luamodules > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/interpreters > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/logging > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/lte > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/math > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/mlearning > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/netutils > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/sdr > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/system > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/tee > > > >>>>>>>>> mkkconfig in > > > >> /home/ligd/platform/mainline/apps/testing/drivers > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/sched > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/libc > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/arch > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/videoutils > > > >>>>>>>>> mkkconfig in > > > >>> /home/ligd/platform/mainline/apps/wireless/bluetooth > > > >>>>>>>>> mkkconfig in > > > >>>> /home/ligd/platform/mainline/apps/wireless/ieee802154 > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/wireless > > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps > > > >>>>>>>>> Loaded configuration '.config' > > > >>>>>>>>> Configuration saved to '.config' > > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > > > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ make* > > > >>>>>>>>> Create version.h > > > >>>>>>>>> Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to > > > >>>>>>>>> /home/ligd/platform/mainline/apps/platform/dummy > > > >>>>>>>>> Downloading: v1.46.0.zip > > > >>>>>>>>> ... > > > >>>>>>>>> patching file > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > > > >>>>>>>>> Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > > > >>>>>>>>> > > > >>>>>>>>> CP: > > > >> /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > > > >>>>>>>>> CP: > > > >>> /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > > > >>>>>>>>> LD: nuttx > > > >>>>>>>>> Pac SIM with dynamic libs.. > > > >>>>>>>>> '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1' > > > >>>>>>>>> '/lib/x86_64-linux-gnu/libstdc++.so.6' -> > > > >>>>>> 'sim-pac/libs/libstdc++.so.6' > > > >>>>>>>>> '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6' > > > >>>>>>>>> '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6' > > > >>>>>>>>> '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> > > > >>>>> 'sim-pac/libs/libgcc_s.so.1' > > > >>>>>>>>> '/lib64/ld-linux-x86-64.so.2' -> > > > >> 'sim-pac/ld-linux-x86-64.so.2' > > > >>>>>>>>> SIM elf with dynamic libs archive in nuttx.tgz > > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > > > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > > > >>>>>>>>> CHelloWorld: Constructor: mSecret=42 > > > >>>>>>>>> nsh: init: open failed: 2 > > > >>>>>>>>> > > > >>>>>>>>> NuttShell (NSH) NuttX-12.11.0 > > > >>>>>>>>> MOTD: username=admin password=Administrator > > > >>>>>>>>> nsh> > > > >>>>>>>>> nsh> help > > > >>>>>>>>> help usage: help [-v] [<cmd>] > > > >>>>>>>>> > > > >>>>>>>>> . df ifconfig mkfifo readlink > > > >>>>>> umount > > > >>>>>>>>> [ dmesg ifdown mkrd rm > > > >>>>> unset > > > >>>>>>>>> ? echo ifup mount rmdir > > > >>>>>> uptime > > > >>>>>>>>> alias env insmod mv rmmod > > > >>>>>> usleep > > > >>>>>>>>> unalias exec kill nslookup set > > > >>>>> watch > > > >>>>>>>>> basename exit pkill pidof sleep > > > >>>> xd > > > >>>>>>>>> break expr losetup pmconfig source > > > >>>>> wait > > > >>>>>>>>> cat false ln poweroff test > > > >>>>>>>>> cd fdinfo ls quit time > > > >>>>>>>>> cp free lsmod printf true > > > >>>>>>>>> cmp help mkdir ps truncate > > > >>>>>>>>> dirname hexdump mkfatfs pwd uname > > > >>>>>>>>> > > > >>>>>>>>> Builtin Apps: > > > >>>>>>>>> cmocka > > > >>>>>>>>> cmocka_driver_block > > > >>>>>>>>> cmocka_driver_gpio > > > >>>>>>>>> cmocka_driver_pm > > > >>>>>>>>> cmocka_driver_pm_runtime > > > >>>>>>>>> cmocka_driver_regulator > > > >>>>>>>>> > > > >>>>>>>>> Matteo Golin <[email protected]> 于2025年12月22日周一 > > > >> 10:15写道： > > > >>>>>>>>> > > > >>>>>>>>>> I think so. Best to leave OS test alone and slowly replace > > > >> it > > > >>>>> with > > > >>>>>> a > > > >>>>>>>> new > > > >>>>>>>>>> tool that has improvements. The only reason I can see to > > > >>> modify > > > >>>>> OS > > > >>>>>>> test > > > >>>>>>>>> now > > > >>>>>>>>>> would just be to have clearer output about what > > > >>> passed/failed. > > > >>>>>>>>>> > > > >>>>>>>>>> I'll come up with an improved proof of concept for a new > > > >> tool > > > >>>> to > > > >>>>>>> share > > > >>>>>>>> in > > > >>>>>>>>>> the next few days. > > > >>>>>>>>>> > > > >>>>>>>>>> Matteo > > > >>>>>>>>>> > > > >>>>>>>>>> On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO < > > > >> [email protected]> > > > >>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Small steps with measurable results may be the best > > > >>> approach > > > >>>>>> here, > > > >>>>>>> to > > > >>>>>>>>>>> create some new solution next to existing ostest but not > > > >>>>>> replacing > > > >>>>>>> it > > > >>>>>>>>>>> until ready? :-) > > > >>>>>>>>>>> > > > >>>>>>>>>>> -- > > > >>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > > > >>>>>>> [email protected] > > > >>>>>>>>> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> I agree, I think it would take a long time to have > > > >> proper > > > >>>>>>> coverage > > > >>>>>>>> of > > > >>>>>>>>>>> NuttX > > > >>>>>>>>>>>> as a whole. Perhaps even impossible, since Unity is > > > >> more > > > >>>> of a > > > >>>>>>> unit > > > >>>>>>>>>>> testing > > > >>>>>>>>>>>> framework and thus there are definitely test cases for > > > >>>> NuttX > > > >>>>>> that > > > >>>>>>>>> would > > > >>>>>>>>>>> be > > > >>>>>>>>>>>> outside of its realm of "applicability". There is > > > >> still a > > > >>>>> need > > > >>>>>>> for > > > >>>>>>>> a > > > >>>>>>>>>>>> variety of testing applications and bench-marking > > > >>>>> applications > > > >>>>>>> for > > > >>>>>>>>>> those > > > >>>>>>>>>>>> scenarios. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> That makes sense, prerequisites would need to be > > > >>>> considered. > > > >>>>> I > > > >>>>>>>> think > > > >>>>>>>>>>>> targeting the low-hanging fruit of unit-testable code > > > >>> would > > > >>>>> be > > > >>>>>>> easy > > > >>>>>>>>> in > > > >>>>>>>>>>> that > > > >>>>>>>>>>>> case, since prerequisites are mostly encoded in the > > > >>> Kconfig > > > >>>>>>>>> definitions > > > >>>>>>>>>>>> existing or not for features to be tested. Any complex > > > >>>>>> behaviour > > > >>>>>>>>>>>> (interrupts, multi-threading, etc.) might be out of the > > > >>>> realm > > > >>>>>> of > > > >>>>>>>>> what's > > > >>>>>>>>>>>> testable by Unity in the first place. We'll always have > > > >>> OS > > > >>>>> test > > > >>>>>>> in > > > >>>>>>>>> that > > > >>>>>>>>>>>> case, but maybe then it's better to do what Alan > > > >>> suggested > > > >>>>> and > > > >>>>>> at > > > >>>>>>>>> least > > > >>>>>>>>>>>> make its output more clear in the meantime. I suppose > > > >>> RTOS > > > >>>>>>> testing > > > >>>>>>>>> is a > > > >>>>>>>>>>>> whole field in itself. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > > > >>>>>>> [email protected]> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> I think that the answer depends on a tradeoff between > > > >>>>> effort > > > >>>>>>> and > > > >>>>>>>>> test > > > >>>>>>>>>>>>> coverage. I could imagine that a proper job could be > > > >>>>>> man-years > > > >>>>>>>> of > > > >>>>>>>>>>> effort. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> My only concern with a re-design is to be certain > > > >> that > > > >>>>> tests > > > >>>>>>> are > > > >>>>>>>>>>>>> arranged so that the prerequisites of later test are > > > >>>> tested > > > >>>>>>>> first. > > > >>>>>>>>>> If > > > >>>>>>>>>>>>> tests are grouped by, for example, functional area, > > > >>> then > > > >>>>>>> failures > > > >>>>>>>>> can > > > >>>>>>>>>>> be > > > >>>>>>>>>>>>> difficult to understand if there is a failure in an > > > >>>>>> unverified > > > >>>>>>>> test > > > >>>>>>>>>>>>> prerequisite. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Some of the "tests" are bogus and only exist to > > > >> provide > > > >>>>> some > > > >>>>>>>>> minimum > > > >>>>>>>>>>>>> confidence that we have the functionality to run the > > > >>>> test. > > > >>>>>> For > > > >>>>>>>>>>>>> example: Can I start tasks? Can I pass parameters? > > > >>> Can > > > >>>> I > > > >>>>>>>> survive > > > >>>>>>>>> a > > > >>>>>>>>>>>>> context switch. Some of the tests were intended only > > > >>> to > > > >>>>>>> support > > > >>>>>>>>>> debug. > > > >>>>>>>>>>>>> Some were created only to track down specific bugs. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On 12/21/25 18:39, Matteo Golin wrote: > > > >>>>>>>>>>>>>> LTS being the Long-Term-Stable release of NuttX? > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> I'm starting to think that it would be best to > > > >> create > > > >>>> new > > > >>>>>>> suite > > > >>>>>>>>> of > > > >>>>>>>>>>> tests > > > >>>>>>>>>>>>>> using Unity, using what's in OS Test as a basis and > > > >>>> then > > > >>>>>>>>> expanding > > > >>>>>>>>>> it > > > >>>>>>>>>>>>>> further. Greg has pointed out that OS test is not > > > >>>>> intended > > > >>>>>> to > > > >>>>>>>> be > > > >>>>>>>>> a > > > >>>>>>>>>>>>> complete > > > >>>>>>>>>>>>>> test, so maybe a more solid testing framework base > > > >>> in a > > > >>>>> new > > > >>>>>>>> form > > > >>>>>>>>>>> would be > > > >>>>>>>>>>>>>> in order. At that point it would also be good to > > > >>>> document > > > >>>>>> and > > > >>>>>>>>>> educate > > > >>>>>>>>>>>>> users > > > >>>>>>>>>>>>>> about other test options, since OS test is a > > > >> current > > > >>>>>>> favourite > > > >>>>>>>>> for > > > >>>>>>>>>>>>> showing > > > >>>>>>>>>>>>>> PRs haven't broken anything. Maybe that's not the > > > >>> best > > > >>>>>> choice > > > >>>>>>>>>> anymore > > > >>>>>>>>>>>>> with > > > >>>>>>>>>>>>>> this information. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> I don't want to re-invent the wheel of what's > > > >> already > > > >>>> in > > > >>>>>>>>>> `testsuite`, > > > >>>>>>>>>>>>> but I > > > >>>>>>>>>>>>>> think Unity might end up being lighter-weight in > > > >> the > > > >>>> long > > > >>>>>>> run? > > > >>>>>>>>> And > > > >>>>>>>>>>> it is > > > >>>>>>>>>>>>>> also not a Xiaomi owned/maintained framework so it > > > >>> may > > > >>>>>>> receive > > > >>>>>>>>>> better > > > >>>>>>>>>>>>>> external support (i.e. more users rely on it?). I > > > >>> tried > > > >>>>> to > > > >>>>>>>>> compile > > > >>>>>>>>>>> the > > > >>>>>>>>>>>>>> sim:citest configuration to run the testsuite > > > >> example > > > >>>> and > > > >>>>>> see > > > >>>>>>>>> what > > > >>>>>>>>>>> it was > > > >>>>>>>>>>>>>> like, but I was met with a long list of C++ > > > >>> compilation > > > >>>>>>> errors > > > >>>>>>>>> that > > > >>>>>>>>>>> I'd > > > >>>>>>>>>>>>>> rather not deal with. There also seems to be zero > > > >>> docs > > > >>>>>> about > > > >>>>>>> it > > > >>>>>>>>> or > > > >>>>>>>>>>> cmocka > > > >>>>>>>>>>>>>> in our NuttX docs at the moment. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> What do you think? Start a Unity test suite, add > > > >> some > > > >>>>>>> switches > > > >>>>>>>> to > > > >>>>>>>>>>> choose > > > >>>>>>>>>>>>>> which test cases get compiled into the binary, open > > > >>> for > > > >>>>>>>>> extensions > > > >>>>>>>>>> by > > > >>>>>>>>>>>>> users > > > >>>>>>>>>>>>>> who know more about different subsystems? This way > > > >> we > > > >>>> can > > > >>>>>>> keep > > > >>>>>>>> OS > > > >>>>>>>>>>> test > > > >>>>>>>>>>>>> how > > > >>>>>>>>>>>>>> it is for a quick, low-resource sanity check of > > > >>> "board > > > >>>>>>> bringup" > > > >>>>>>>>>> while > > > >>>>>>>>>>>>>> having a dedicated testing application for patches? > > > >>>>> Someone > > > >>>>>>>>>> changing > > > >>>>>>>>>>> file > > > >>>>>>>>>>>>>> system stuff can just compile to include file > > > >> system > > > >>>> test > > > >>>>>>> cases > > > >>>>>>>>> and > > > >>>>>>>>>>> run > > > >>>>>>>>>>>>>> that suite only as proof that their PR works. The > > > >>>> output > > > >>>>> is > > > >>>>>>>>> really > > > >>>>>>>>>>> easy > > > >>>>>>>>>>>>> to > > > >>>>>>>>>>>>>> glance at to determine pass/fail results. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Anyways, here's my draft demonstrating how Unity > > > >>> works > > > >>>>> with > > > >>>>>>> the > > > >>>>>>>>> FPU > > > >>>>>>>>>>> test > > > >>>>>>>>>>>>>> suite from OS test: > > > >>>>>>>>> https://github.com/apache/nuttx-apps/pull/3259 > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Matteo > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > > > >>>>>>> [email protected]> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > > > >>>>>>>>> [email protected] > > > >>>>>>>>>>> > > > >>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>> (..) > > > >>>>>>>>>>>>>>>> Starting with bits and pieces of something like > > > >> the > > > >>>> LTS > > > >>>>>>> might > > > >>>>>>>>>> make > > > >>>>>>>>>>> more > > > >>>>>>>>>>>>>>>> sense other than its licensing problems. Since > > > >> it > > > >>>>> would > > > >>>>>>>> never > > > >>>>>>>>> be > > > >>>>>>>>>>>>>>>> included in a distributed binary, the licensing > > > >>>> really > > > >>>>>>> should > > > >>>>>>>>> not > > > >>>>>>>>>>>>> matter > > > >>>>>>>>>>>>>>>> much in practice. > > > >>>>>>>>>>>>>>> Unless its GPL and some company would like to have > > > >>>> their > > > >>>>>> own > > > >>>>>>>>>>> internal > > > >>>>>>>>>>>>>>> bits and pieces.. luckily its MIT and the whole > > > >>> thing > > > >>>> is > > > >>>>>>>>>> open-source > > > >>>>>>>>>>>>>>> anyways :-) > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Yes we had many discussions about the LTS and this > > > >>>> could > > > >>>>>> be > > > >>>>>>>>> great > > > >>>>>>>>>>> start! > > > >>>>>>>>>>>>>>> :-) > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> -- > > > >>>>>>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > >
