On Tue, Dec 23, 2025 at 10:29 AM Matteo Golin <[email protected]> wrote:
> Thank you Xiang for the information! It makes sense of course that you're > not able to share the Xiaomi internal test suites. > > Actually, raiden00pl is working on upstreaming the part of the test suite which tests NuttX's components to the NuttX community. The part we can't share is that: 1. The test suite specific for openvela framework 2. Internal CI/CD infrastructure which is couple with our workflow and very different from github > Are there any modifications Xiaomi has needed to make to cmocka to improve > it's ability to run on small devices? I know some of your targets are > wearables which I imagine are very resource constrained? > > No, we don't do any size optimization for cmocka since it is already a small unit test framework containing one .h and one .c. We run many test cases(including cmocka's) on the smallest BLE only module which has 256KB SRAM and 2MB flash. > Can you clarify what you mean by monkey test? Is that another test > framework? I'm interested since you mentioned it running on watches and > speakers. > > Here is an explanation from wiki: https://en.wikipedia.org/wiki/Monkey_testing https://en.wikipedia.org/wiki/Mean_time_between_failures Here is the our monkey test requirement: 1. Boot 50 instances for each simulator/qemu/hardware with the product config 2. Generate the random input quickly through the fake drivers(e.g. touch/button/audio...) All devices need to run beyond 24 hours without crash or hang for three consecutive days. Thanks, > Matteo > > On Mon, Dec 22, 2025, 9:08 PM Xiang Xiao <[email protected]> > wrote: > > > On Tue, Dec 23, 2025 at 1:08 AM Matteo Golin <[email protected]> > > wrote: > > > > > Hello everyone, > > > > > > Thanks for all the feedback! I'm glad I discussed this here first. I'll > > try > > > to address everything that was brought up. > > > > > > Thanks Guiding for the input! I must just be missing some library on my > > > machine if you're able to compile it just fine. Maybe I'll track down > > that > > > error and see if you can recognize what's wrong. > > > > > > I don't intend to replace OS test anymore. I think it serves its > purpose > > as > > > a minimal sanity check. Instead, I think we need something a little > more > > > involved to test PRs since OS test can't catch many issues. It is > > currently > > > the standard everyone uses for verifying their patches and it would be > > > better to have more complete, alternate options. I do think that we can > > all > > > agree that OS test needs a revision of its console output so that it is > > > much easier to see pass/fail results, but that is just a matter of > print > > > statements and can be done in a different effort. > > > > > > I think the overwhelming concern here is mixing two unit testing > > frameworks > > > and no longer having one resource where we can go for tests. I share > > raiden > > > and Guiding's concern about that, which is why I'm glad I opened for > > > discussion here. On the one hand, Xiaomi has a pretty substantial > > > collection of tests in the `testsuite` application which they are using > > > internally I believe (in their CI). These run well on virtual machines, > > > which I assume is why the only configurations with that test suite > > included > > > is sim and rv-virt. I do not know nearly enough about cmocka to make > any > > > claim that it *wouldn't* work on smaller hardware devices, but it also > > > definitely looks to be less light-weight from the feature set in the > > > documentation. On the other hand, the Xiaomi test suites are also not a > > > complete silver bullet since they seem not to be intended to run on > > > hardware, and there are *a lot *of tests that need hardware > verification. > > > > > > > The test suite also runs on hardware daily, but sorry it's impossible to > > share our hardware and internal CI/CD infrastructure for the community. > > But, we already contract raiden00pl to refactor our test automation > > framework for public use. > > > > > > > In PRs we often request more than just a simulator test for certain > > > changes. For instance, sim isn't able to have `/dev/rand`, so that > needs > > to > > > be tested on a different target. > > > > > > > Maybe someone more familiar with Xiaomi's testing practices can weigh in > on > > > whether cmocka has been run on hardware. Going purely based off of > their > > > PRs, which almost exclusively cite tests on simulators, my guess is > that > > > cmocka was not designed with embedded in mind when it was written. > > > > > > > > CI/CD in Xiaomi is far more complex than your description: > > > > 1. Each patch will test on 10+ devices(include sim/qemu/hardware, > > arm/arm64/riscv/xtensa/x64, smp/up) with 1000+ test case > > 2. Run the full test suite(10000+ test case) every day > > 3. Run the monkey test every day on watch and speaker product: 24hour > X > > 50 instance X (sim + qemu + hardware) > > > > Many stability bugs only happen randomly in monkey test. > > > > As for size: unit testing can get large with Unity or cmocka (as raiden > > > mentioned) as more tests are added. Hence my effort to enable/disable > > > individual test suites from getting compiled. It might not be possible > to > > > ever get good testing ability for AVR just because the devices are so > > > small. Whatever solution is used there is probably going to involve > > > flashing the device multiple times with different tests. The size goal > > > should just be to target the most NuttX devices possible. > > > > > > Then there are the pros and cons of each framework (again, I don't know > > > much about cmocka so take it with a grain of salt): > > > > > > Unity's cons: > > > - Designed for each test-suite to be a standalone binary. This is > clunky > > on > > > NuttX, so `nest` is currently implemented as a mega-suite where test > > cases > > > just follow a naming convention to make it clear what is being tested > > > - setUp/tearDown can't really be leveraged because of the above > > > - I've noticed some funky behaviour using TEST_IGNORE in certain > > scenarios > > > but that's possibly my own configuration issue > > > - Would be in conflict with existing cmocka tests > > > > > > Unity's pros: > > > - Highly configurable, like NuttX > > > - Light-weight and works for embedded devices > > > - Significantly larger user base than cmocka > > > > > > Cmocka's cons: > > > - Possibly (?) not light-weight > > > - No documentation for existing test cases in NuttX > > > - Small user base > > > > > > Cmocka's pros: > > > - Seems to have a larger set of features, including setup/teardown > > > functions per-test case > > > - Existing tests in NuttX > > > > > > > > Xiaomi selected cmocka just because it has the test suite concept which > > could help us manage 10000+ tests. > > But we also use gtest for the c++ code base, and run many test cases from > > 3rd parties(e.g. LTP, libuv, libcxx testsuite, PSE52...) which may use a > > different test framework. > > So, is it hard or meaningless to stick to one unit test framework for an > > open source project. > > > > Irrespective of whether we go with Unity or cmocka, the collection of > tests > > > that we write will be intended to run on the actual hardware devices > > > themselves for the purposes of testing patches and doing regression > > tests. > > > As a result, the tests will be separate from the Xiaomi 'testsuite' > > > application which is intended for CI testing. I think this > realistically > > > nullifies choosing which framework to use on the basis of consistency, > > > since the tests are not coexisting with the cmocka tests Xiaomi has > > > written. Instead, the choice should be based on what best achieves the > > goal > > > from a functionality perspective. Unity seems like a really great > choice > > > for this in my opinion, but I am bothered by the fact that it's > designed > > > with multiple test binaries in mind. > > > > > > Let me know what you think, > > > Matteo > > > > > > On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <[email protected]> > > wrote: > > > > > > > Yes, I agree! We need data size from each solution for comparison, > this > > > is > > > > the first step. > > > > > > > > Maybe the conversion could be facilitated if we create equivalent > > > functions > > > > like those used in the apps/testing/testsuites: > > > > > > > > cmocka_run_group_tests -> unity_run_group_tests > > > > cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown > > > > etc... > > > > > > > > This way we avoid end up with two unit test framework, we already > have > > > the > > > > Makefile vs CMake precedent! :-D > > > > > > > > On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <[email protected]> > > > wrote: > > > > > > > > > The discussion about small systems doesn't make much sense without > > > > > data comparing resources of the same test cases for cmock and > unity. > > > > > > > > > > But let's assume that Unity is better. Should we use two frameworks > > > > > to implement unit tests for OS? Or choose one framework? If we > choose > > > > only > > > > > Unity, who > > > > > will migrate the current cmocka tests to Unity? Remember that cmoka > > > tests > > > > > are used in CI, so all CI tools must be also migrated. > > > > > Or should we use two different frameworks to test different things > > and > > > > make > > > > > even > > > > > more mess in apps/testing as it is now? > > > > > > > > > > No one is forcing anyone to do anything. This is an open-source > > > project, > > > > > everyone > > > > > should do what they feel is right. But whether the change should be > > > > merged > > > > > is > > > > > a different question. In this case, it's a very important question, > > > > because > > > > > unit testing > > > > > isn't just about test cases. It's about the entire infrastructure > > that > > > > > runs, captures and > > > > > handles test results later. Writing the test cases is the least of > > the > > > > > problems. > > > > > The infrastructure for cmoka testing is already in place, the > > > > > infrastructure for Unity > > > > > testing isn't. > > > > > > > > > > pon., 22 gru 2025 o 14:16 Alan C. Assis <[email protected]> > > > napisał(a): > > > > > > > > > > > Mateusz, > > > > > > but some chips like AVR DA have big flash, but not so much RAM. > > > > > > > > > > > > It is better to have some possible way to test on a small system, > > > using > > > > > > cmocka we are removing this option from the table. > > > > > > > > > > > > I liked the test Matteo did with Unity and I think we should let > > him > > > > > > continue. Forcing him to migrate to cmocka just because there's a > > > test > > > > > > suite from Xiaomi doesn't seem prudent to me. > > > > > > > > > > > > This is the advantage of an open mind open-source project: "the > > > > > possibility > > > > > > to test new ideas", if someone had said to Michal Lenc he > couldn't > > > > create > > > > > > NXBoot because where is already MCUBoot integrated on NuttX, we > > > didn't > > > > > have > > > > > > the option to use NXBoot today. ;-) > > > > > > > > > > > > BR, > > > > > > > > > > > > Alan > > > > > > > > > > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > Unit testing by itself isn't suitable for small systems. Ostest > > is > > > > > > > something > > > > > > > different (system level testing), so running it on small > systems > > > > makes > > > > > > > sense. > > > > > > > Running unit tests on small systems is pointless because you > > won't > > > > fit > > > > > in > > > > > > > FLASH > > > > > > > either way. You can run the same on large targets (preferably > on > > > > host: > > > > > > sim, > > > > > > > qemu) > > > > > > > and achieve the same result. > > > > > > > > > > > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <[email protected]> > > > > > napisał(a): > > > > > > > > > > > > > > > Is it too complicated to rewrite the cmocka tests in Unity? > > > > > > > > > > > > > > > > If cmoka is not feasible for small microcontrollers, like > AVRs > > > and > > > > > > > others, > > > > > > > > our test coverage will be faulty. > > > > > > > > > > > > > > > > BR, > > > > > > > > > > > > > > > > Alan > > > > > > > > > > > > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I'm a fan of Unity and have been using it in my projects > for > > > > years, > > > > > > > but I > > > > > > > > > think > > > > > > > > > we should stick with one testing framework. Currently, > Cmocka > > > is > > > > > used > > > > > > > in > > > > > > > > > nuttx-apps, and we should stick with it. The only question > is > > > how > > > > > > well > > > > > > > > > Cmocka > > > > > > > > > is suitable for running on small systems? Will ostest still > > be > > > > able > > > > > > to > > > > > > > > run > > > > > > > > > on small MCUs? > > > > > > > > > With Unity this would be quite likely, so maybe an > exception > > > for > > > > > > Unity > > > > > > > > and > > > > > > > > > ostest would > > > > > > > > > be acceptable, but using it for other test cases will only > > > cause > > > > > > chaos. > > > > > > > > > > > > > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> > > > > > napisał(a): > > > > > > > > > > > > > > > > > > > Hi Matteo, > > > > > > > > > > > > > > > > > > > > Good idea ! > > > > > > > > > > > > > > > > > > > > But it is best to stick with a single testing framework. > > > > > Currently, > > > > > > > > many > > > > > > > > > > test cases are using cmocka, so I suggest reusing cmocka > > for > > > > > > > > consistency. > > > > > > > > > > > > > > > > > > > > ostest is a minimal basic test suite, typically used for > CT > > > > > > > (continuous > > > > > > > > > > testing) of each patch. > > > > > > > > > > If we move ostest to cmocka, it would make ostest > dependent > > > on > > > > a > > > > > > > > specific > > > > > > > > > > framework before it can be run. > > > > > > > > > > And It may increase the code size, which is not very > > friendly > > > > for > > > > > > > > > > configurations with small flash memory. > > > > > > > > > > > > > > > > > > > > As Greg mentioned: the answer depends on a tradeoff > between > > > > > effort > > > > > > > and > > > > > > > > > test > > > > > > > > > > coverage. > > > > > > > > > > > > > > > > > > > > We need to clarify whether we define ostest as a > > lightweight, > > > > > > > > > ready-to-run > > > > > > > > > > test set for quick verification, or as part of the full > > test > > > > > suite. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The follow is my compiling for cmocka: > > > > > > > > > > > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ > ./tools/configure.sh > > > > > > sim:citest > > > > > > > > > -j8* > > > > > > > > > > Copy files > > > > > > > > > > Select CONFIG_HOST_LINUX=y > > > > > > > > > > Refreshing... > > > > > > > > > > CP: arch/dummy/Kconfig to > > > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > > > > > > > > > > CP: boards/dummy/Kconfig to > > > > > > > > > > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > > > > > > > > > > LN: include/arch to arch/sim/include > > > > > > > > > > LN: drivers/platform to > > > > > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy > > > > > > > > > > LN: arch/sim/src/chip to > > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > > > > > > > > > > LN: arch/sim/src/board to > > > > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > > > > > > > > > > LN: include/arch/board to > > > > > > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > > > > > > > > > > LN: include/arch/chip to > > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > > > > > > > > > > LN: platform/board to > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database > > > > > > > > > > mkkconfig in > > /home/ligd/platform/mainline/apps/examples/rust > > > > > > > > > > mkkconfig in > > > /home/ligd/platform/mainline/apps/examples/sotest > > > > > > > > > > mkkconfig in > > > /home/ligd/platform/mainline/apps/examples/module > > > > > > > > > > mkkconfig in > > > /home/ligd/platform/mainline/apps/examples/mcuboot > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial > > > > > > > > > > mkkconfig in > > > > > > > /home/ligd/platform/mainline/apps/interpreters/luamodules > > > > > > > > > > mkkconfig in > /home/ligd/platform/mainline/apps/interpreters > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee > > > > > > > > > > mkkconfig in > > > /home/ligd/platform/mainline/apps/testing/drivers > > > > > > > > > > mkkconfig in > > /home/ligd/platform/mainline/apps/testing/sched > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > > > > > > > > > > mkkconfig in > /home/ligd/platform/mainline/apps/testing/libc > > > > > > > > > > mkkconfig in > /home/ligd/platform/mainline/apps/testing/arch > > > > > > > > > > mkkconfig in > /home/ligd/platform/mainline/apps/testing/cxx > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils > > > > > > > > > > mkkconfig in > > > > /home/ligd/platform/mainline/apps/wireless/bluetooth > > > > > > > > > > mkkconfig in > > > > > /home/ligd/platform/mainline/apps/wireless/ieee802154 > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps > > > > > > > > > > Loaded configuration '.config' > > > > > > > > > > Configuration saved to '.config' > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make* > > > > > > > > > > Create version.h > > > > > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board > to > > > > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy > > > > > > > > > > Downloading: v1.46.0.zip > > > > > > > > > > ... > > > > > > > > > > patching file > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > > > > > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > > > > > > > > > > > > > > > > > > > > CP: > > > /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > > > > > > > > > > CP: > > > > /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > > > > > > > > > > LD: nuttx > > > > > > > > > > Pac SIM with dynamic libs.. > > > > > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> > > 'sim-pac/libs/libz.so.1' > > > > > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> > > > > > > > 'sim-pac/libs/libstdc++.so.6' > > > > > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> > > 'sim-pac/libs/libc.so.6' > > > > > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> > > 'sim-pac/libs/libm.so.6' > > > > > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> > > > > > > 'sim-pac/libs/libgcc_s.so.1' > > > > > > > > > > '/lib64/ld-linux-x86-64.so.2' -> > > > 'sim-pac/ld-linux-x86-64.so.2' > > > > > > > > > > SIM elf with dynamic libs archive in nuttx.tgz > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > > > > > > > > > > CHelloWorld: Constructor: mSecret=42 > > > > > > > > > > nsh: init: open failed: 2 > > > > > > > > > > > > > > > > > > > > NuttShell (NSH) NuttX-12.11.0 > > > > > > > > > > MOTD: username=admin password=Administrator > > > > > > > > > > nsh> > > > > > > > > > > nsh> help > > > > > > > > > > help usage: help [-v] [<cmd>] > > > > > > > > > > > > > > > > > > > > . df ifconfig mkfifo > > readlink > > > > > > > umount > > > > > > > > > > [ dmesg ifdown mkrd rm > > > > > > unset > > > > > > > > > > ? echo ifup mount rmdir > > > > > > > uptime > > > > > > > > > > alias env insmod mv rmmod > > > > > > > usleep > > > > > > > > > > unalias exec kill nslookup set > > > > > > watch > > > > > > > > > > basename exit pkill pidof sleep > > > > > xd > > > > > > > > > > break expr losetup pmconfig > source > > > > > > wait > > > > > > > > > > cat false ln poweroff test > > > > > > > > > > cd fdinfo ls quit time > > > > > > > > > > cp free lsmod printf true > > > > > > > > > > cmp help mkdir ps > > truncate > > > > > > > > > > dirname hexdump mkfatfs pwd uname > > > > > > > > > > > > > > > > > > > > Builtin Apps: > > > > > > > > > > cmocka > > > > > > > > > > cmocka_driver_block > > > > > > > > > > cmocka_driver_gpio > > > > > > > > > > cmocka_driver_pm > > > > > > > > > > cmocka_driver_pm_runtime > > > > > > > > > > cmocka_driver_regulator > > > > > > > > > > > > > > > > > > > > Matteo Golin <[email protected]> 于2025年12月22日周一 > > > 10:15写道： > > > > > > > > > > > > > > > > > > > > > I think so. Best to leave OS test alone and slowly > > replace > > > it > > > > > > with > > > > > > > a > > > > > > > > > new > > > > > > > > > > > tool that has improvements. The only reason I can see > to > > > > modify > > > > > > OS > > > > > > > > test > > > > > > > > > > now > > > > > > > > > > > would just be to have clearer output about what > > > > passed/failed. > > > > > > > > > > > > > > > > > > > > > > I'll come up with an improved proof of concept for a > new > > > tool > > > > > to > > > > > > > > share > > > > > > > > > in > > > > > > > > > > > the next few days. > > > > > > > > > > > > > > > > > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Small steps with measurable results may be the best > > > > approach > > > > > > > here, > > > > > > > > to > > > > > > > > > > > > create some new solution next to existing ostest but > > not > > > > > > > replacing > > > > > > > > it > > > > > > > > > > > > until ready? :-) > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > I agree, I think it would take a long time to have > > > proper > > > > > > > > coverage > > > > > > > > > of > > > > > > > > > > > > NuttX > > > > > > > > > > > > > as a whole. Perhaps even impossible, since Unity is > > > more > > > > > of a > > > > > > > > unit > > > > > > > > > > > > testing > > > > > > > > > > > > > framework and thus there are definitely test cases > > for > > > > > NuttX > > > > > > > that > > > > > > > > > > would > > > > > > > > > > > > be > > > > > > > > > > > > > outside of its realm of "applicability". There is > > > still a > > > > > > need > > > > > > > > for > > > > > > > > > a > > > > > > > > > > > > > variety of testing applications and bench-marking > > > > > > applications > > > > > > > > for > > > > > > > > > > > those > > > > > > > > > > > > > scenarios. > > > > > > > > > > > > > > > > > > > > > > > > > > That makes sense, prerequisites would need to be > > > > > considered. > > > > > > I > > > > > > > > > think > > > > > > > > > > > > > targeting the low-hanging fruit of unit-testable > code > > > > would > > > > > > be > > > > > > > > easy > > > > > > > > > > in > > > > > > > > > > > > that > > > > > > > > > > > > > case, since prerequisites are mostly encoded in the > > > > Kconfig > > > > > > > > > > definitions > > > > > > > > > > > > > existing or not for features to be tested. Any > > complex > > > > > > > behaviour > > > > > > > > > > > > > (interrupts, multi-threading, etc.) might be out of > > the > > > > > realm > > > > > > > of > > > > > > > > > > what's > > > > > > > > > > > > > testable by Unity in the first place. We'll always > > have > > > > OS > > > > > > test > > > > > > > > in > > > > > > > > > > that > > > > > > > > > > > > > case, but maybe then it's better to do what Alan > > > > suggested > > > > > > and > > > > > > > at > > > > > > > > > > least > > > > > > > > > > > > > make its output more clear in the meantime. I > suppose > > > > RTOS > > > > > > > > testing > > > > > > > > > > is a > > > > > > > > > > > > > whole field in itself. > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I think that the answer depends on a tradeoff > > between > > > > > > effort > > > > > > > > and > > > > > > > > > > test > > > > > > > > > > > > > > coverage. I could imagine that a proper job > could > > be > > > > > > > man-years > > > > > > > > > of > > > > > > > > > > > > effort. > > > > > > > > > > > > > > > > > > > > > > > > > > > > My only concern with a re-design is to be certain > > > that > > > > > > tests > > > > > > > > are > > > > > > > > > > > > > > arranged so that the prerequisites of later test > > are > > > > > tested > > > > > > > > > first. > > > > > > > > > > > If > > > > > > > > > > > > > > tests are grouped by, for example, functional > area, > > > > then > > > > > > > > failures > > > > > > > > > > can > > > > > > > > > > > > be > > > > > > > > > > > > > > difficult to understand if there is a failure in > an > > > > > > > unverified > > > > > > > > > test > > > > > > > > > > > > > > prerequisite. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Some of the "tests" are bogus and only exist to > > > provide > > > > > > some > > > > > > > > > > minimum > > > > > > > > > > > > > > confidence that we have the functionality to run > > the > > > > > test. > > > > > > > For > > > > > > > > > > > > > > example: Can I start tasks? Can I pass > > parameters? > > > > Can > > > > > I > > > > > > > > > survive > > > > > > > > > > a > > > > > > > > > > > > > > context switch. Some of the tests were intended > > only > > > > to > > > > > > > > support > > > > > > > > > > > debug. > > > > > > > > > > > > > > Some were created only to track down specific > bugs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote: > > > > > > > > > > > > > > > LTS being the Long-Term-Stable release of > NuttX? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm starting to think that it would be best to > > > create > > > > > new > > > > > > > > suite > > > > > > > > > > of > > > > > > > > > > > > tests > > > > > > > > > > > > > > > using Unity, using what's in OS Test as a basis > > and > > > > > then > > > > > > > > > > expanding > > > > > > > > > > > it > > > > > > > > > > > > > > > further. Greg has pointed out that OS test is > not > > > > > > intended > > > > > > > to > > > > > > > > > be > > > > > > > > > > a > > > > > > > > > > > > > > complete > > > > > > > > > > > > > > > test, so maybe a more solid testing framework > > base > > > > in a > > > > > > new > > > > > > > > > form > > > > > > > > > > > > would be > > > > > > > > > > > > > > > in order. At that point it would also be good > to > > > > > document > > > > > > > and > > > > > > > > > > > educate > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > about other test options, since OS test is a > > > current > > > > > > > > favourite > > > > > > > > > > for > > > > > > > > > > > > > > showing > > > > > > > > > > > > > > > PRs haven't broken anything. Maybe that's not > the > > > > best > > > > > > > choice > > > > > > > > > > > anymore > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > this information. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't want to re-invent the wheel of what's > > > already > > > > > in > > > > > > > > > > > `testsuite`, > > > > > > > > > > > > > > but I > > > > > > > > > > > > > > > think Unity might end up being lighter-weight > in > > > the > > > > > long > > > > > > > > run? > > > > > > > > > > And > > > > > > > > > > > > it is > > > > > > > > > > > > > > > also not a Xiaomi owned/maintained framework so > > it > > > > may > > > > > > > > receive > > > > > > > > > > > better > > > > > > > > > > > > > > > external support (i.e. more users rely on > it?). I > > > > tried > > > > > > to > > > > > > > > > > compile > > > > > > > > > > > > the > > > > > > > > > > > > > > > sim:citest configuration to run the testsuite > > > example > > > > > and > > > > > > > see > > > > > > > > > > what > > > > > > > > > > > > it was > > > > > > > > > > > > > > > like, but I was met with a long list of C++ > > > > compilation > > > > > > > > errors > > > > > > > > > > that > > > > > > > > > > > > I'd > > > > > > > > > > > > > > > rather not deal with. There also seems to be > zero > > > > docs > > > > > > > about > > > > > > > > it > > > > > > > > > > or > > > > > > > > > > > > cmocka > > > > > > > > > > > > > > > in our NuttX docs at the moment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What do you think? Start a Unity test suite, > add > > > some > > > > > > > > switches > > > > > > > > > to > > > > > > > > > > > > choose > > > > > > > > > > > > > > > which test cases get compiled into the binary, > > open > > > > for > > > > > > > > > > extensions > > > > > > > > > > > by > > > > > > > > > > > > > > users > > > > > > > > > > > > > > > who know more about different subsystems? This > > way > > > we > > > > > can > > > > > > > > keep > > > > > > > > > OS > > > > > > > > > > > > test > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > it is for a quick, low-resource sanity check of > > > > "board > > > > > > > > bringup" > > > > > > > > > > > while > > > > > > > > > > > > > > > having a dedicated testing application for > > patches? > > > > > > Someone > > > > > > > > > > > changing > > > > > > > > > > > > file > > > > > > > > > > > > > > > system stuff can just compile to include file > > > system > > > > > test > > > > > > > > cases > > > > > > > > > > and > > > > > > > > > > > > run > > > > > > > > > > > > > > > that suite only as proof that their PR works. > The > > > > > output > > > > > > is > > > > > > > > > > really > > > > > > > > > > > > easy > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > glance at to determine pass/fail results. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Anyways, here's my draft demonstrating how > Unity > > > > works > > > > > > with > > > > > > > > the > > > > > > > > > > FPU > > > > > > > > > > > > test > > > > > > > > > > > > > > > suite from OS test: > > > > > > > > > > https://github.com/apache/nuttx-apps/pull/3259 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >>> (..) > > > > > > > > > > > > > > >>> Starting with bits and pieces of something > like > > > the > > > > > LTS > > > > > > > > might > > > > > > > > > > > make > > > > > > > > > > > > more > > > > > > > > > > > > > > >>> sense other than its licensing problems. > Since > > > it > > > > > > would > > > > > > > > > never > > > > > > > > > > be > > > > > > > > > > > > > > >>> included in a distributed binary, the > licensing > > > > > really > > > > > > > > should > > > > > > > > > > not > > > > > > > > > > > > > > matter > > > > > > > > > > > > > > >>> much in practice. > > > > > > > > > > > > > > >> Unless its GPL and some company would like to > > have > > > > > their > > > > > > > own > > > > > > > > > > > > internal > > > > > > > > > > > > > > >> bits and pieces.. luckily its MIT and the > whole > > > > thing > > > > > is > > > > > > > > > > > open-source > > > > > > > > > > > > > > >> anyways :-) > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Yes we had many discussions about the LTS and > > this > > > > > could > > > > > > > be > > > > > > > > > > great > > > > > > > > > > > > start! > > > > > > > > > > > > > > >> :-) > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> -- > > > > > > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
