Hi Pierre-Noel, The instructions are here: https://nuttx.apache.org/community/
We are glad for all the things you did for NuttX (EFM32 port, ESP8266, etc.) Thank you very much! BR, Alan On Mon, Dec 22, 2025 at 3:16 PM Pierre-Noel Bouteville <[email protected]> wrote: > How to unsbcribe? > Pierre-Noël Bouteville > Envoyé de mon iPhone > > > Le 22 déc. 2025 à 19:13, raiden00pl <[email protected]> a écrit : > > > > Cmocka tests can run on any target. It doesn't matter if it's sim, qemu > or > > real hardware. > > You just need to prepare a defconfig and have the right amount of FLASH. > > > > For small systems we can always try to reduce the size of the cmock, e.g. > > by disabling > > unnecessary prints. I doubt that the logic of running tests in cmoka is > > heavy, running > > test cases are not a complicated operation, I think most of the required > > FLASH is wasted > > on strings. > > > > NTFC supports serial port communication and sample test scenario based on > > nucleo-h743z > > with cmocka tests enabled. The only thing missing for full test > automation > > (compile, flash, > > test, result) is cmake support for testsuites. > > > > pon., 22 gru 2025 o 18:08 Matteo Golin <[email protected]> > napisał(a): > > > >> Hello everyone, > >> > >> Thanks for all the feedback! I'm glad I discussed this here first. I'll > try > >> to address everything that was brought up. > >> > >> Thanks Guiding for the input! I must just be missing some library on my > >> machine if you're able to compile it just fine. Maybe I'll track down > that > >> error and see if you can recognize what's wrong. > >> > >> I don't intend to replace OS test anymore. I think it serves its > purpose as > >> a minimal sanity check. Instead, I think we need something a little more > >> involved to test PRs since OS test can't catch many issues. It is > currently > >> the standard everyone uses for verifying their patches and it would be > >> better to have more complete, alternate options. I do think that we can > all > >> agree that OS test needs a revision of its console output so that it is > >> much easier to see pass/fail results, but that is just a matter of print > >> statements and can be done in a different effort. > >> > >> I think the overwhelming concern here is mixing two unit testing > frameworks > >> and no longer having one resource where we can go for tests. I share > raiden > >> and Guiding's concern about that, which is why I'm glad I opened for > >> discussion here. On the one hand, Xiaomi has a pretty substantial > >> collection of tests in the `testsuite` application which they are using > >> internally I believe (in their CI). These run well on virtual machines, > >> which I assume is why the only configurations with that test suite > included > >> is sim and rv-virt. I do not know nearly enough about cmocka to make any > >> claim that it *wouldn't* work on smaller hardware devices, but it also > >> definitely looks to be less light-weight from the feature set in the > >> documentation. On the other hand, the Xiaomi test suites are also not a > >> complete silver bullet since they seem not to be intended to run on > >> hardware, and there are *a lot *of tests that need hardware > verification. > >> In PRs we often request more than just a simulator test for certain > >> changes. For instance, sim isn't able to have `/dev/rand`, so that > needs to > >> be tested on a different target. > >> > >> Maybe someone more familiar with Xiaomi's testing practices can weigh > in on > >> whether cmocka has been run on hardware. Going purely based off of their > >> PRs, which almost exclusively cite tests on simulators, my guess is that > >> cmocka was not designed with embedded in mind when it was written. > >> > >> As for size: unit testing can get large with Unity or cmocka (as raiden > >> mentioned) as more tests are added. Hence my effort to enable/disable > >> individual test suites from getting compiled. It might not be possible > to > >> ever get good testing ability for AVR just because the devices are so > >> small. Whatever solution is used there is probably going to involve > >> flashing the device multiple times with different tests. The size goal > >> should just be to target the most NuttX devices possible. > >> > >> Then there are the pros and cons of each framework (again, I don't know > >> much about cmocka so take it with a grain of salt): > >> > >> Unity's cons: > >> - Designed for each test-suite to be a standalone binary. This is > clunky on > >> NuttX, so `nest` is currently implemented as a mega-suite where test > cases > >> just follow a naming convention to make it clear what is being tested > >> - setUp/tearDown can't really be leveraged because of the above > >> - I've noticed some funky behaviour using TEST_IGNORE in certain > scenarios > >> but that's possibly my own configuration issue > >> - Would be in conflict with existing cmocka tests > >> > >> Unity's pros: > >> - Highly configurable, like NuttX > >> - Light-weight and works for embedded devices > >> - Significantly larger user base than cmocka > >> > >> Cmocka's cons: > >> - Possibly (?) not light-weight > >> - No documentation for existing test cases in NuttX > >> - Small user base > >> > >> Cmocka's pros: > >> - Seems to have a larger set of features, including setup/teardown > >> functions per-test case > >> - Existing tests in NuttX > >> > >> Irrespective of whether we go with Unity or cmocka, the collection of > tests > >> that we write will be intended to run on the actual hardware devices > >> themselves for the purposes of testing patches and doing regression > tests. > >> As a result, the tests will be separate from the Xiaomi 'testsuite' > >> application which is intended for CI testing. I think this realistically > >> nullifies choosing which framework to use on the basis of consistency, > >> since the tests are not coexisting with the cmocka tests Xiaomi has > >> written. Instead, the choice should be based on what best achieves the > goal > >> from a functionality perspective. Unity seems like a really great choice > >> for this in my opinion, but I am bothered by the fact that it's designed > >> with multiple test binaries in mind. > >> > >> Let me know what you think, > >> Matteo > >> > >>> On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <[email protected]> > wrote: > >>> > >>> Yes, I agree! We need data size from each solution for comparison, this > >> is > >>> the first step. > >>> > >>> Maybe the conversion could be facilitated if we create equivalent > >> functions > >>> like those used in the apps/testing/testsuites: > >>> > >>> cmocka_run_group_tests -> unity_run_group_tests > >>> cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown > >>> etc... > >>> > >>> This way we avoid end up with two unit test framework, we already have > >> the > >>> Makefile vs CMake precedent! :-D > >>> > >>> On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <[email protected]> > >> wrote: > >>> > >>>> The discussion about small systems doesn't make much sense without > >>>> data comparing resources of the same test cases for cmock and unity. > >>>> > >>>> But let's assume that Unity is better. Should we use two frameworks > >>>> to implement unit tests for OS? Or choose one framework? If we choose > >>> only > >>>> Unity, who > >>>> will migrate the current cmocka tests to Unity? Remember that cmoka > >> tests > >>>> are used in CI, so all CI tools must be also migrated. > >>>> Or should we use two different frameworks to test different things and > >>> make > >>>> even > >>>> more mess in apps/testing as it is now? > >>>> > >>>> No one is forcing anyone to do anything. This is an open-source > >> project, > >>>> everyone > >>>> should do what they feel is right. But whether the change should be > >>> merged > >>>> is > >>>> a different question. In this case, it's a very important question, > >>> because > >>>> unit testing > >>>> isn't just about test cases. It's about the entire infrastructure that > >>>> runs, captures and > >>>> handles test results later. Writing the test cases is the least of the > >>>> problems. > >>>> The infrastructure for cmoka testing is already in place, the > >>>> infrastructure for Unity > >>>> testing isn't. > >>>> > >>>> pon., 22 gru 2025 o 14:16 Alan C. Assis <[email protected]> > >> napisał(a): > >>>> > >>>>> Mateusz, > >>>>> but some chips like AVR DA have big flash, but not so much RAM. > >>>>> > >>>>> It is better to have some possible way to test on a small system, > >> using > >>>>> cmocka we are removing this option from the table. > >>>>> > >>>>> I liked the test Matteo did with Unity and I think we should let him > >>>>> continue. Forcing him to migrate to cmocka just because there's a > >> test > >>>>> suite from Xiaomi doesn't seem prudent to me. > >>>>> > >>>>> This is the advantage of an open mind open-source project: "the > >>>> possibility > >>>>> to test new ideas", if someone had said to Michal Lenc he couldn't > >>> create > >>>>> NXBoot because where is already MCUBoot integrated on NuttX, we > >> didn't > >>>> have > >>>>> the option to use NXBoot today. ;-) > >>>>> > >>>>> BR, > >>>>> > >>>>> Alan > >>>>> > >>>>> On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <[email protected]> > >>>> wrote: > >>>>> > >>>>>> Unit testing by itself isn't suitable for small systems. Ostest is > >>>>>> something > >>>>>> different (system level testing), so running it on small systems > >>> makes > >>>>>> sense. > >>>>>> Running unit tests on small systems is pointless because you won't > >>> fit > >>>> in > >>>>>> FLASH > >>>>>> either way. You can run the same on large targets (preferably on > >>> host: > >>>>> sim, > >>>>>> qemu) > >>>>>> and achieve the same result. > >>>>>> > >>>>>> pon., 22 gru 2025 o 13:35 Alan C. Assis <[email protected]> > >>>> napisał(a): > >>>>>> > >>>>>>> Is it too complicated to rewrite the cmocka tests in Unity? > >>>>>>> > >>>>>>> If cmoka is not feasible for small microcontrollers, like AVRs > >> and > >>>>>> others, > >>>>>>> our test coverage will be faulty. > >>>>>>> > >>>>>>> BR, > >>>>>>> > >>>>>>> Alan > >>>>>>> > >>>>>>> On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <[email protected] > >>> > >>>>> wrote: > >>>>>>> > >>>>>>>> I'm a fan of Unity and have been using it in my projects for > >>> years, > >>>>>> but I > >>>>>>>> think > >>>>>>>> we should stick with one testing framework. Currently, Cmocka > >> is > >>>> used > >>>>>> in > >>>>>>>> nuttx-apps, and we should stick with it. The only question is > >> how > >>>>> well > >>>>>>>> Cmocka > >>>>>>>> is suitable for running on small systems? Will ostest still be > >>> able > >>>>> to > >>>>>>> run > >>>>>>>> on small MCUs? > >>>>>>>> With Unity this would be quite likely, so maybe an exception > >> for > >>>>> Unity > >>>>>>> and > >>>>>>>> ostest would > >>>>>>>> be acceptable, but using it for other test cases will only > >> cause > >>>>> chaos. > >>>>>>>> > >>>>>>>> pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> > >>>> napisał(a): > >>>>>>>> > >>>>>>>>> Hi Matteo, > >>>>>>>>> > >>>>>>>>> Good idea ! > >>>>>>>>> > >>>>>>>>> But it is best to stick with a single testing framework. > >>>> Currently, > >>>>>>> many > >>>>>>>>> test cases are using cmocka, so I suggest reusing cmocka for > >>>>>>> consistency. > >>>>>>>>> > >>>>>>>>> ostest is a minimal basic test suite, typically used for CT > >>>>>> (continuous > >>>>>>>>> testing) of each patch. > >>>>>>>>> If we move ostest to cmocka, it would make ostest dependent > >> on > >>> a > >>>>>>> specific > >>>>>>>>> framework before it can be run. > >>>>>>>>> And It may increase the code size, which is not very friendly > >>> for > >>>>>>>>> configurations with small flash memory. > >>>>>>>>> > >>>>>>>>> As Greg mentioned: the answer depends on a tradeoff between > >>>> effort > >>>>>> and > >>>>>>>> test > >>>>>>>>> coverage. > >>>>>>>>> > >>>>>>>>> We need to clarify whether we define ostest as a lightweight, > >>>>>>>> ready-to-run > >>>>>>>>> test set for quick verification, or as part of the full test > >>>> suite. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> The follow is my compiling for cmocka: > >>>>>>>>> > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh > >>>>> sim:citest > >>>>>>>> -j8* > >>>>>>>>> Copy files > >>>>>>>>> Select CONFIG_HOST_LINUX=y > >>>>>>>>> Refreshing... > >>>>>>>>> CP: arch/dummy/Kconfig to > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > >>>>>>>>> CP: boards/dummy/Kconfig to > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > >>>>>>>>> LN: include/arch to arch/sim/include > >>>>>>>>> LN: drivers/platform to > >>>>>>> /home/ligd/platform/mainline/nuttx/drivers/dummy > >>>>>>>>> LN: arch/sim/src/chip to > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > >>>>>>>>> LN: arch/sim/src/board to > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > >>>>>>>>> LN: include/arch/board to > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > >>>>>>>>> LN: include/arch/chip to > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > >>>>>>>>> LN: platform/board to > >>>>>> /home/ligd/platform/mainline/apps/platform/dummy > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/audioutils > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/boot > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/canutils > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/crypto > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/database > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples/rust > >>>>>>>>> mkkconfig in > >> /home/ligd/platform/mainline/apps/examples/sotest > >>>>>>>>> mkkconfig in > >> /home/ligd/platform/mainline/apps/examples/module > >>>>>>>>> mkkconfig in > >> /home/ligd/platform/mainline/apps/examples/mcuboot > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/fsutils > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/games > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/graphics > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/industry > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/inertial > >>>>>>>>> mkkconfig in > >>>>>> /home/ligd/platform/mainline/apps/interpreters/luamodules > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/interpreters > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/logging > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/lte > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/math > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/mlearning > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/netutils > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/sdr > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/system > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/tee > >>>>>>>>> mkkconfig in > >> /home/ligd/platform/mainline/apps/testing/drivers > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/sched > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/libc > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/arch > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/videoutils > >>>>>>>>> mkkconfig in > >>> /home/ligd/platform/mainline/apps/wireless/bluetooth > >>>>>>>>> mkkconfig in > >>>> /home/ligd/platform/mainline/apps/wireless/ieee802154 > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/wireless > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps > >>>>>>>>> Loaded configuration '.config' > >>>>>>>>> Configuration saved to '.config' > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ make* > >>>>>>>>> Create version.h > >>>>>>>>> Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to > >>>>>>>>> /home/ligd/platform/mainline/apps/platform/dummy > >>>>>>>>> Downloading: v1.46.0.zip > >>>>>>>>> ... > >>>>>>>>> patching file > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > >>>>>>>>> Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > >>>>>>>>> > >>>>>>>>> CP: > >> /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > >>>>>>>>> CP: > >>> /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > >>>>>>>>> LD: nuttx > >>>>>>>>> Pac SIM with dynamic libs.. > >>>>>>>>> '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1' > >>>>>>>>> '/lib/x86_64-linux-gnu/libstdc++.so.6' -> > >>>>>> 'sim-pac/libs/libstdc++.so.6' > >>>>>>>>> '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6' > >>>>>>>>> '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6' > >>>>>>>>> '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> > >>>>> 'sim-pac/libs/libgcc_s.so.1' > >>>>>>>>> '/lib64/ld-linux-x86-64.so.2' -> > >> 'sim-pac/ld-linux-x86-64.so.2' > >>>>>>>>> SIM elf with dynamic libs archive in nuttx.tgz > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$ > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > >>>>>>>>> CHelloWorld: Constructor: mSecret=42 > >>>>>>>>> nsh: init: open failed: 2 > >>>>>>>>> > >>>>>>>>> NuttShell (NSH) NuttX-12.11.0 > >>>>>>>>> MOTD: username=admin password=Administrator > >>>>>>>>> nsh> > >>>>>>>>> nsh> help > >>>>>>>>> help usage: help [-v] [<cmd>] > >>>>>>>>> > >>>>>>>>> . df ifconfig mkfifo readlink > >>>>>> umount > >>>>>>>>> [ dmesg ifdown mkrd rm > >>>>> unset > >>>>>>>>> ? echo ifup mount rmdir > >>>>>> uptime > >>>>>>>>> alias env insmod mv rmmod > >>>>>> usleep > >>>>>>>>> unalias exec kill nslookup set > >>>>> watch > >>>>>>>>> basename exit pkill pidof sleep > >>>> xd > >>>>>>>>> break expr losetup pmconfig source > >>>>> wait > >>>>>>>>> cat false ln poweroff test > >>>>>>>>> cd fdinfo ls quit time > >>>>>>>>> cp free lsmod printf true > >>>>>>>>> cmp help mkdir ps truncate > >>>>>>>>> dirname hexdump mkfatfs pwd uname > >>>>>>>>> > >>>>>>>>> Builtin Apps: > >>>>>>>>> cmocka > >>>>>>>>> cmocka_driver_block > >>>>>>>>> cmocka_driver_gpio > >>>>>>>>> cmocka_driver_pm > >>>>>>>>> cmocka_driver_pm_runtime > >>>>>>>>> cmocka_driver_regulator > >>>>>>>>> > >>>>>>>>> Matteo Golin <[email protected]> 于2025年12月22日周一 > >> 10:15写道： > >>>>>>>>> > >>>>>>>>>> I think so. Best to leave OS test alone and slowly replace > >> it > >>>>> with > >>>>>> a > >>>>>>>> new > >>>>>>>>>> tool that has improvements. The only reason I can see to > >>> modify > >>>>> OS > >>>>>>> test > >>>>>>>>> now > >>>>>>>>>> would just be to have clearer output about what > >>> passed/failed. > >>>>>>>>>> > >>>>>>>>>> I'll come up with an improved proof of concept for a new > >> tool > >>>> to > >>>>>>> share > >>>>>>>> in > >>>>>>>>>> the next few days. > >>>>>>>>>> > >>>>>>>>>> Matteo > >>>>>>>>>> > >>>>>>>>>> On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO < > >> [email protected]> > >>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Small steps with measurable results may be the best > >>> approach > >>>>>> here, > >>>>>>> to > >>>>>>>>>>> create some new solution next to existing ostest but not > >>>>>> replacing > >>>>>>> it > >>>>>>>>>>> until ready? :-) > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > >>>>>>> [email protected] > >>>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> I agree, I think it would take a long time to have > >> proper > >>>>>>> coverage > >>>>>>>> of > >>>>>>>>>>> NuttX > >>>>>>>>>>>> as a whole. Perhaps even impossible, since Unity is > >> more > >>>> of a > >>>>>>> unit > >>>>>>>>>>> testing > >>>>>>>>>>>> framework and thus there are definitely test cases for > >>>> NuttX > >>>>>> that > >>>>>>>>> would > >>>>>>>>>>> be > >>>>>>>>>>>> outside of its realm of "applicability". There is > >> still a > >>>>> need > >>>>>>> for > >>>>>>>> a > >>>>>>>>>>>> variety of testing applications and bench-marking > >>>>> applications > >>>>>>> for > >>>>>>>>>> those > >>>>>>>>>>>> scenarios. > >>>>>>>>>>>> > >>>>>>>>>>>> That makes sense, prerequisites would need to be > >>>> considered. > >>>>> I > >>>>>>>> think > >>>>>>>>>>>> targeting the low-hanging fruit of unit-testable code > >>> would > >>>>> be > >>>>>>> easy > >>>>>>>>> in > >>>>>>>>>>> that > >>>>>>>>>>>> case, since prerequisites are mostly encoded in the > >>> Kconfig > >>>>>>>>> definitions > >>>>>>>>>>>> existing or not for features to be tested. Any complex > >>>>>> behaviour > >>>>>>>>>>>> (interrupts, multi-threading, etc.) might be out of the > >>>> realm > >>>>>> of > >>>>>>>>> what's > >>>>>>>>>>>> testable by Unity in the first place. We'll always have > >>> OS > >>>>> test > >>>>>>> in > >>>>>>>>> that > >>>>>>>>>>>> case, but maybe then it's better to do what Alan > >>> suggested > >>>>> and > >>>>>> at > >>>>>>>>> least > >>>>>>>>>>>> make its output more clear in the meantime. I suppose > >>> RTOS > >>>>>>> testing > >>>>>>>>> is a > >>>>>>>>>>>> whole field in itself. > >>>>>>>>>>>> > >>>>>>>>>>>> On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > >>>>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> I think that the answer depends on a tradeoff between > >>>>> effort > >>>>>>> and > >>>>>>>>> test > >>>>>>>>>>>>> coverage. I could imagine that a proper job could be > >>>>>> man-years > >>>>>>>> of > >>>>>>>>>>> effort. > >>>>>>>>>>>>> > >>>>>>>>>>>>> My only concern with a re-design is to be certain > >> that > >>>>> tests > >>>>>>> are > >>>>>>>>>>>>> arranged so that the prerequisites of later test are > >>>> tested > >>>>>>>> first. > >>>>>>>>>> If > >>>>>>>>>>>>> tests are grouped by, for example, functional area, > >>> then > >>>>>>> failures > >>>>>>>>> can > >>>>>>>>>>> be > >>>>>>>>>>>>> difficult to understand if there is a failure in an > >>>>>> unverified > >>>>>>>> test > >>>>>>>>>>>>> prerequisite. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Some of the "tests" are bogus and only exist to > >> provide > >>>>> some > >>>>>>>>> minimum > >>>>>>>>>>>>> confidence that we have the functionality to run the > >>>> test. > >>>>>> For > >>>>>>>>>>>>> example: Can I start tasks? Can I pass parameters? > >>> Can > >>>> I > >>>>>>>> survive > >>>>>>>>> a > >>>>>>>>>>>>> context switch. Some of the tests were intended only > >>> to > >>>>>>> support > >>>>>>>>>> debug. > >>>>>>>>>>>>> Some were created only to track down specific bugs. > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 12/21/25 18:39, Matteo Golin wrote: > >>>>>>>>>>>>>> LTS being the Long-Term-Stable release of NuttX? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I'm starting to think that it would be best to > >> create > >>>> new > >>>>>>> suite > >>>>>>>>> of > >>>>>>>>>>> tests > >>>>>>>>>>>>>> using Unity, using what's in OS Test as a basis and > >>>> then > >>>>>>>>> expanding > >>>>>>>>>> it > >>>>>>>>>>>>>> further. Greg has pointed out that OS test is not > >>>>> intended > >>>>>> to > >>>>>>>> be > >>>>>>>>> a > >>>>>>>>>>>>> complete > >>>>>>>>>>>>>> test, so maybe a more solid testing framework base > >>> in a > >>>>> new > >>>>>>>> form > >>>>>>>>>>> would be > >>>>>>>>>>>>>> in order. At that point it would also be good to > >>>> document > >>>>>> and > >>>>>>>>>> educate > >>>>>>>>>>>>> users > >>>>>>>>>>>>>> about other test options, since OS test is a > >> current > >>>>>>> favourite > >>>>>>>>> for > >>>>>>>>>>>>> showing > >>>>>>>>>>>>>> PRs haven't broken anything. Maybe that's not the > >>> best > >>>>>> choice > >>>>>>>>>> anymore > >>>>>>>>>>>>> with > >>>>>>>>>>>>>> this information. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I don't want to re-invent the wheel of what's > >> already > >>>> in > >>>>>>>>>> `testsuite`, > >>>>>>>>>>>>> but I > >>>>>>>>>>>>>> think Unity might end up being lighter-weight in > >> the > >>>> long > >>>>>>> run? > >>>>>>>>> And > >>>>>>>>>>> it is > >>>>>>>>>>>>>> also not a Xiaomi owned/maintained framework so it > >>> may > >>>>>>> receive > >>>>>>>>>> better > >>>>>>>>>>>>>> external support (i.e. more users rely on it?). I > >>> tried > >>>>> to > >>>>>>>>> compile > >>>>>>>>>>> the > >>>>>>>>>>>>>> sim:citest configuration to run the testsuite > >> example > >>>> and > >>>>>> see > >>>>>>>>> what > >>>>>>>>>>> it was > >>>>>>>>>>>>>> like, but I was met with a long list of C++ > >>> compilation > >>>>>>> errors > >>>>>>>>> that > >>>>>>>>>>> I'd > >>>>>>>>>>>>>> rather not deal with. There also seems to be zero > >>> docs > >>>>>> about > >>>>>>> it > >>>>>>>>> or > >>>>>>>>>>> cmocka > >>>>>>>>>>>>>> in our NuttX docs at the moment. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> What do you think? Start a Unity test suite, add > >> some > >>>>>>> switches > >>>>>>>> to > >>>>>>>>>>> choose > >>>>>>>>>>>>>> which test cases get compiled into the binary, open > >>> for > >>>>>>>>> extensions > >>>>>>>>>> by > >>>>>>>>>>>>> users > >>>>>>>>>>>>>> who know more about different subsystems? This way > >> we > >>>> can > >>>>>>> keep > >>>>>>>> OS > >>>>>>>>>>> test > >>>>>>>>>>>>> how > >>>>>>>>>>>>>> it is for a quick, low-resource sanity check of > >>> "board > >>>>>>> bringup" > >>>>>>>>>> while > >>>>>>>>>>>>>> having a dedicated testing application for patches? > >>>>> Someone > >>>>>>>>>> changing > >>>>>>>>>>> file > >>>>>>>>>>>>>> system stuff can just compile to include file > >> system > >>>> test > >>>>>>> cases > >>>>>>>>> and > >>>>>>>>>>> run > >>>>>>>>>>>>>> that suite only as proof that their PR works. The > >>>> output > >>>>> is > >>>>>>>>> really > >>>>>>>>>>> easy > >>>>>>>>>>>>> to > >>>>>>>>>>>>>> glance at to determine pass/fail results. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Anyways, here's my draft demonstrating how Unity > >>> works > >>>>> with > >>>>>>> the > >>>>>>>>> FPU > >>>>>>>>>>> test > >>>>>>>>>>>>>> suite from OS test: > >>>>>>>>> https://github.com/apache/nuttx-apps/pull/3259 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Matteo > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > >>>>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > >>>>>>>>> [email protected] > >>>>>>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> (..) > >>>>>>>>>>>>>>>> Starting with bits and pieces of something like > >> the > >>>> LTS > >>>>>>> might > >>>>>>>>>> make > >>>>>>>>>>> more > >>>>>>>>>>>>>>>> sense other than its licensing problems. Since > >> it > >>>>> would > >>>>>>>> never > >>>>>>>>> be > >>>>>>>>>>>>>>>> included in a distributed binary, the licensing > >>>> really > >>>>>>> should > >>>>>>>>> not > >>>>>>>>>>>>> matter > >>>>>>>>>>>>>>>> much in practice. > >>>>>>>>>>>>>>> Unless its GPL and some company would like to have > >>>> their > >>>>>> own > >>>>>>>>>>> internal > >>>>>>>>>>>>>>> bits and pieces.. luckily its MIT and the whole > >>> thing > >>>> is > >>>>>>>>>> open-source > >>>>>>>>>>>>>>> anyways :-) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Yes we had many discussions about the LTS and this > >>>> could > >>>>>> be > >>>>>>>>> great > >>>>>>>>>>> start! > >>>>>>>>>>>>>>> :-) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> >
