Thank you Xiang for the information! It makes sense of course that you're not able to share the Xiaomi internal test suites.
Are there any modifications Xiaomi has needed to make to cmocka to improve it's ability to run on small devices? I know some of your targets are wearables which I imagine are very resource constrained? Can you clarify what you mean by monkey test? Is that another test framework? I'm interested since you mentioned it running on watches and speakers. Thanks, Matteo On Mon, Dec 22, 2025, 9:08 PM Xiang Xiao <[email protected]> wrote: > On Tue, Dec 23, 2025 at 1:08 AM Matteo Golin <[email protected]> > wrote: > > > Hello everyone, > > > > Thanks for all the feedback! I'm glad I discussed this here first. I'll > try > > to address everything that was brought up. > > > > Thanks Guiding for the input! I must just be missing some library on my > > machine if you're able to compile it just fine. Maybe I'll track down > that > > error and see if you can recognize what's wrong. > > > > I don't intend to replace OS test anymore. I think it serves its purpose > as > > a minimal sanity check. Instead, I think we need something a little more > > involved to test PRs since OS test can't catch many issues. It is > currently > > the standard everyone uses for verifying their patches and it would be > > better to have more complete, alternate options. I do think that we can > all > > agree that OS test needs a revision of its console output so that it is > > much easier to see pass/fail results, but that is just a matter of print > > statements and can be done in a different effort. > > > > I think the overwhelming concern here is mixing two unit testing > frameworks > > and no longer having one resource where we can go for tests. I share > raiden > > and Guiding's concern about that, which is why I'm glad I opened for > > discussion here. On the one hand, Xiaomi has a pretty substantial > > collection of tests in the `testsuite` application which they are using > > internally I believe (in their CI). These run well on virtual machines, > > which I assume is why the only configurations with that test suite > included > > is sim and rv-virt. I do not know nearly enough about cmocka to make any > > claim that it *wouldn't* work on smaller hardware devices, but it also > > definitely looks to be less light-weight from the feature set in the > > documentation. On the other hand, the Xiaomi test suites are also not a > > complete silver bullet since they seem not to be intended to run on > > hardware, and there are *a lot *of tests that need hardware verification. > > > > The test suite also runs on hardware daily, but sorry it's impossible to > share our hardware and internal CI/CD infrastructure for the community. > But, we already contract raiden00pl to refactor our test automation > framework for public use. > > > > In PRs we often request more than just a simulator test for certain > > changes. For instance, sim isn't able to have `/dev/rand`, so that needs > to > > be tested on a different target. > > > > Maybe someone more familiar with Xiaomi's testing practices can weigh in on > > whether cmocka has been run on hardware. Going purely based off of their > > PRs, which almost exclusively cite tests on simulators, my guess is that > > cmocka was not designed with embedded in mind when it was written. > > > > > CI/CD in Xiaomi is far more complex than your description: > > 1. Each patch will test on 10+ devices(include sim/qemu/hardware, > arm/arm64/riscv/xtensa/x64, smp/up) with 1000+ test case > 2. Run the full test suite(10000+ test case) every day > 3. Run the monkey test every day on watch and speaker product: 24hour X > 50 instance X (sim + qemu + hardware) > > Many stability bugs only happen randomly in monkey test. > > As for size: unit testing can get large with Unity or cmocka (as raiden > > mentioned) as more tests are added. Hence my effort to enable/disable > > individual test suites from getting compiled. It might not be possible to > > ever get good testing ability for AVR just because the devices are so > > small. Whatever solution is used there is probably going to involve > > flashing the device multiple times with different tests. The size goal > > should just be to target the most NuttX devices possible. > > > > Then there are the pros and cons of each framework (again, I don't know > > much about cmocka so take it with a grain of salt): > > > > Unity's cons: > > - Designed for each test-suite to be a standalone binary. This is clunky > on > > NuttX, so `nest` is currently implemented as a mega-suite where test > cases > > just follow a naming convention to make it clear what is being tested > > - setUp/tearDown can't really be leveraged because of the above > > - I've noticed some funky behaviour using TEST_IGNORE in certain > scenarios > > but that's possibly my own configuration issue > > - Would be in conflict with existing cmocka tests > > > > Unity's pros: > > - Highly configurable, like NuttX > > - Light-weight and works for embedded devices > > - Significantly larger user base than cmocka > > > > Cmocka's cons: > > - Possibly (?) not light-weight > > - No documentation for existing test cases in NuttX > > - Small user base > > > > Cmocka's pros: > > - Seems to have a larger set of features, including setup/teardown > > functions per-test case > > - Existing tests in NuttX > > > > > Xiaomi selected cmocka just because it has the test suite concept which > could help us manage 10000+ tests. > But we also use gtest for the c++ code base, and run many test cases from > 3rd parties(e.g. LTP, libuv, libcxx testsuite, PSE52...) which may use a > different test framework. > So, is it hard or meaningless to stick to one unit test framework for an > open source project. > > Irrespective of whether we go with Unity or cmocka, the collection of tests > > that we write will be intended to run on the actual hardware devices > > themselves for the purposes of testing patches and doing regression > tests. > > As a result, the tests will be separate from the Xiaomi 'testsuite' > > application which is intended for CI testing. I think this realistically > > nullifies choosing which framework to use on the basis of consistency, > > since the tests are not coexisting with the cmocka tests Xiaomi has > > written. Instead, the choice should be based on what best achieves the > goal > > from a functionality perspective. Unity seems like a really great choice > > for this in my opinion, but I am bothered by the fact that it's designed > > with multiple test binaries in mind. > > > > Let me know what you think, > > Matteo > > > > On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <[email protected]> > wrote: > > > > > Yes, I agree! We need data size from each solution for comparison, this > > is > > > the first step. > > > > > > Maybe the conversion could be facilitated if we create equivalent > > functions > > > like those used in the apps/testing/testsuites: > > > > > > cmocka_run_group_tests -> unity_run_group_tests > > > cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown > > > etc... > > > > > > This way we avoid end up with two unit test framework, we already have > > the > > > Makefile vs CMake precedent! :-D > > > > > > On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <[email protected]> > > wrote: > > > > > > > The discussion about small systems doesn't make much sense without > > > > data comparing resources of the same test cases for cmock and unity. > > > > > > > > But let's assume that Unity is better. Should we use two frameworks > > > > to implement unit tests for OS? Or choose one framework? If we choose > > > only > > > > Unity, who > > > > will migrate the current cmocka tests to Unity? Remember that cmoka > > tests > > > > are used in CI, so all CI tools must be also migrated. > > > > Or should we use two different frameworks to test different things > and > > > make > > > > even > > > > more mess in apps/testing as it is now? > > > > > > > > No one is forcing anyone to do anything. This is an open-source > > project, > > > > everyone > > > > should do what they feel is right. But whether the change should be > > > merged > > > > is > > > > a different question. In this case, it's a very important question, > > > because > > > > unit testing > > > > isn't just about test cases. It's about the entire infrastructure > that > > > > runs, captures and > > > > handles test results later. Writing the test cases is the least of > the > > > > problems. > > > > The infrastructure for cmoka testing is already in place, the > > > > infrastructure for Unity > > > > testing isn't. > > > > > > > > pon., 22 gru 2025 o 14:16 Alan C. Assis <[email protected]> > > napisał(a): > > > > > > > > > Mateusz, > > > > > but some chips like AVR DA have big flash, but not so much RAM. > > > > > > > > > > It is better to have some possible way to test on a small system, > > using > > > > > cmocka we are removing this option from the table. > > > > > > > > > > I liked the test Matteo did with Unity and I think we should let > him > > > > > continue. Forcing him to migrate to cmocka just because there's a > > test > > > > > suite from Xiaomi doesn't seem prudent to me. > > > > > > > > > > This is the advantage of an open mind open-source project: "the > > > > possibility > > > > > to test new ideas", if someone had said to Michal Lenc he couldn't > > > create > > > > > NXBoot because where is already MCUBoot integrated on NuttX, we > > didn't > > > > have > > > > > the option to use NXBoot today. ;-) > > > > > > > > > > BR, > > > > > > > > > > Alan > > > > > > > > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <[email protected]> > > > > wrote: > > > > > > > > > > > Unit testing by itself isn't suitable for small systems. Ostest > is > > > > > > something > > > > > > different (system level testing), so running it on small systems > > > makes > > > > > > sense. > > > > > > Running unit tests on small systems is pointless because you > won't > > > fit > > > > in > > > > > > FLASH > > > > > > either way. You can run the same on large targets (preferably on > > > host: > > > > > sim, > > > > > > qemu) > > > > > > and achieve the same result. > > > > > > > > > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <[email protected]> > > > > napisał(a): > > > > > > > > > > > > > Is it too complicated to rewrite the cmocka tests in Unity? > > > > > > > > > > > > > > If cmoka is not feasible for small microcontrollers, like AVRs > > and > > > > > > others, > > > > > > > our test coverage will be faulty. > > > > > > > > > > > > > > BR, > > > > > > > > > > > > > > Alan > > > > > > > > > > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > I'm a fan of Unity and have been using it in my projects for > > > years, > > > > > > but I > > > > > > > > think > > > > > > > > we should stick with one testing framework. Currently, Cmocka > > is > > > > used > > > > > > in > > > > > > > > nuttx-apps, and we should stick with it. The only question is > > how > > > > > well > > > > > > > > Cmocka > > > > > > > > is suitable for running on small systems? Will ostest still > be > > > able > > > > > to > > > > > > > run > > > > > > > > on small MCUs? > > > > > > > > With Unity this would be quite likely, so maybe an exception > > for > > > > > Unity > > > > > > > and > > > > > > > > ostest would > > > > > > > > be acceptable, but using it for other test cases will only > > cause > > > > > chaos. > > > > > > > > > > > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> > > > > napisał(a): > > > > > > > > > > > > > > > > > Hi Matteo, > > > > > > > > > > > > > > > > > > Good idea ! > > > > > > > > > > > > > > > > > > But it is best to stick with a single testing framework. > > > > Currently, > > > > > > > many > > > > > > > > > test cases are using cmocka, so I suggest reusing cmocka > for > > > > > > > consistency. > > > > > > > > > > > > > > > > > > ostest is a minimal basic test suite, typically used for CT > > > > > > (continuous > > > > > > > > > testing) of each patch. > > > > > > > > > If we move ostest to cmocka, it would make ostest dependent > > on > > > a > > > > > > > specific > > > > > > > > > framework before it can be run. > > > > > > > > > And It may increase the code size, which is not very > friendly > > > for > > > > > > > > > configurations with small flash memory. > > > > > > > > > > > > > > > > > > As Greg mentioned: the answer depends on a tradeoff between > > > > effort > > > > > > and > > > > > > > > test > > > > > > > > > coverage. > > > > > > > > > > > > > > > > > > We need to clarify whether we define ostest as a > lightweight, > > > > > > > > ready-to-run > > > > > > > > > test set for quick verification, or as part of the full > test > > > > suite. > > > > > > > > > > > > > > > > > > > > > > > > > > > The follow is my compiling for cmocka: > > > > > > > > > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh > > > > > sim:citest > > > > > > > > -j8* > > > > > > > > > Copy files > > > > > > > > > Select CONFIG_HOST_LINUX=y > > > > > > > > > Refreshing... > > > > > > > > > CP: arch/dummy/Kconfig to > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig > > > > > > > > > CP: boards/dummy/Kconfig to > > > > > > > > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig > > > > > > > > > LN: include/arch to arch/sim/include > > > > > > > > > LN: drivers/platform to > > > > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy > > > > > > > > > LN: arch/sim/src/chip to > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim > > > > > > > > > LN: arch/sim/src/board to > > > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src > > > > > > > > > LN: include/arch/board to > > > > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include > > > > > > > > > LN: include/arch/chip to > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim > > > > > > > > > LN: platform/board to > > > > > > /home/ligd/platform/mainline/apps/platform/dummy > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database > > > > > > > > > mkkconfig in > /home/ligd/platform/mainline/apps/examples/rust > > > > > > > > > mkkconfig in > > /home/ligd/platform/mainline/apps/examples/sotest > > > > > > > > > mkkconfig in > > /home/ligd/platform/mainline/apps/examples/module > > > > > > > > > mkkconfig in > > /home/ligd/platform/mainline/apps/examples/mcuboot > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial > > > > > > > > > mkkconfig in > > > > > > /home/ligd/platform/mainline/apps/interpreters/luamodules > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee > > > > > > > > > mkkconfig in > > /home/ligd/platform/mainline/apps/testing/drivers > > > > > > > > > mkkconfig in > /home/ligd/platform/mainline/apps/testing/sched > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils > > > > > > > > > mkkconfig in > > > /home/ligd/platform/mainline/apps/wireless/bluetooth > > > > > > > > > mkkconfig in > > > > /home/ligd/platform/mainline/apps/wireless/ieee802154 > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps > > > > > > > > > Loaded configuration '.config' > > > > > > > > > Configuration saved to '.config' > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make* > > > > > > > > > Create version.h > > > > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to > > > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy > > > > > > > > > Downloading: v1.46.0.zip > > > > > > > > > ... > > > > > > > > > patching file > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c > > > > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines). > > > > > > > > > > > > > > > > > > CP: > > /home/ligd/platform/mainline/nuttx/include/nuttx/config.h > > > > > > > > > CP: > > > /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h > > > > > > > > > LD: nuttx > > > > > > > > > Pac SIM with dynamic libs.. > > > > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> > 'sim-pac/libs/libz.so.1' > > > > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> > > > > > > 'sim-pac/libs/libstdc++.so.6' > > > > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> > 'sim-pac/libs/libc.so.6' > > > > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> > 'sim-pac/libs/libm.so.6' > > > > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> > > > > > 'sim-pac/libs/libgcc_s.so.1' > > > > > > > > > '/lib64/ld-linux-x86-64.so.2' -> > > 'sim-pac/ld-linux-x86-64.so.2' > > > > > > > > > SIM elf with dynamic libs archive in nuttx.tgz > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * > > > > > > > > > CHelloWorld: Constructor: mSecret=42 > > > > > > > > > nsh: init: open failed: 2 > > > > > > > > > > > > > > > > > > NuttShell (NSH) NuttX-12.11.0 > > > > > > > > > MOTD: username=admin password=Administrator > > > > > > > > > nsh> > > > > > > > > > nsh> help > > > > > > > > > help usage: help [-v] [<cmd>] > > > > > > > > > > > > > > > > > > . df ifconfig mkfifo > readlink > > > > > > umount > > > > > > > > > [ dmesg ifdown mkrd rm > > > > > unset > > > > > > > > > ? echo ifup mount rmdir > > > > > > uptime > > > > > > > > > alias env insmod mv rmmod > > > > > > usleep > > > > > > > > > unalias exec kill nslookup set > > > > > watch > > > > > > > > > basename exit pkill pidof sleep > > > > xd > > > > > > > > > break expr losetup pmconfig source > > > > > wait > > > > > > > > > cat false ln poweroff test > > > > > > > > > cd fdinfo ls quit time > > > > > > > > > cp free lsmod printf true > > > > > > > > > cmp help mkdir ps > truncate > > > > > > > > > dirname hexdump mkfatfs pwd uname > > > > > > > > > > > > > > > > > > Builtin Apps: > > > > > > > > > cmocka > > > > > > > > > cmocka_driver_block > > > > > > > > > cmocka_driver_gpio > > > > > > > > > cmocka_driver_pm > > > > > > > > > cmocka_driver_pm_runtime > > > > > > > > > cmocka_driver_regulator > > > > > > > > > > > > > > > > > > Matteo Golin <[email protected]> 于2025年12月22日周一 > > 10:15写道： > > > > > > > > > > > > > > > > > > > I think so. Best to leave OS test alone and slowly > replace > > it > > > > > with > > > > > > a > > > > > > > > new > > > > > > > > > > tool that has improvements. The only reason I can see to > > > modify > > > > > OS > > > > > > > test > > > > > > > > > now > > > > > > > > > > would just be to have clearer output about what > > > passed/failed. > > > > > > > > > > > > > > > > > > > > I'll come up with an improved proof of concept for a new > > tool > > > > to > > > > > > > share > > > > > > > > in > > > > > > > > > > the next few days. > > > > > > > > > > > > > > > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Small steps with measurable results may be the best > > > approach > > > > > > here, > > > > > > > to > > > > > > > > > > > create some new solution next to existing ostest but > not > > > > > > replacing > > > > > > > it > > > > > > > > > > > until ready? :-) > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > I agree, I think it would take a long time to have > > proper > > > > > > > coverage > > > > > > > > of > > > > > > > > > > > NuttX > > > > > > > > > > > > as a whole. Perhaps even impossible, since Unity is > > more > > > > of a > > > > > > > unit > > > > > > > > > > > testing > > > > > > > > > > > > framework and thus there are definitely test cases > for > > > > NuttX > > > > > > that > > > > > > > > > would > > > > > > > > > > > be > > > > > > > > > > > > outside of its realm of "applicability". There is > > still a > > > > > need > > > > > > > for > > > > > > > > a > > > > > > > > > > > > variety of testing applications and bench-marking > > > > > applications > > > > > > > for > > > > > > > > > > those > > > > > > > > > > > > scenarios. > > > > > > > > > > > > > > > > > > > > > > > > That makes sense, prerequisites would need to be > > > > considered. > > > > > I > > > > > > > > think > > > > > > > > > > > > targeting the low-hanging fruit of unit-testable code > > > would > > > > > be > > > > > > > easy > > > > > > > > > in > > > > > > > > > > > that > > > > > > > > > > > > case, since prerequisites are mostly encoded in the > > > Kconfig > > > > > > > > > definitions > > > > > > > > > > > > existing or not for features to be tested. Any > complex > > > > > > behaviour > > > > > > > > > > > > (interrupts, multi-threading, etc.) might be out of > the > > > > realm > > > > > > of > > > > > > > > > what's > > > > > > > > > > > > testable by Unity in the first place. We'll always > have > > > OS > > > > > test > > > > > > > in > > > > > > > > > that > > > > > > > > > > > > case, but maybe then it's better to do what Alan > > > suggested > > > > > and > > > > > > at > > > > > > > > > least > > > > > > > > > > > > make its output more clear in the meantime. I suppose > > > RTOS > > > > > > > testing > > > > > > > > > is a > > > > > > > > > > > > whole field in itself. > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > I think that the answer depends on a tradeoff > between > > > > > effort > > > > > > > and > > > > > > > > > test > > > > > > > > > > > > > coverage. I could imagine that a proper job could > be > > > > > > man-years > > > > > > > > of > > > > > > > > > > > effort. > > > > > > > > > > > > > > > > > > > > > > > > > > My only concern with a re-design is to be certain > > that > > > > > tests > > > > > > > are > > > > > > > > > > > > > arranged so that the prerequisites of later test > are > > > > tested > > > > > > > > first. > > > > > > > > > > If > > > > > > > > > > > > > tests are grouped by, for example, functional area, > > > then > > > > > > > failures > > > > > > > > > can > > > > > > > > > > > be > > > > > > > > > > > > > difficult to understand if there is a failure in an > > > > > > unverified > > > > > > > > test > > > > > > > > > > > > > prerequisite. > > > > > > > > > > > > > > > > > > > > > > > > > > Some of the "tests" are bogus and only exist to > > provide > > > > > some > > > > > > > > > minimum > > > > > > > > > > > > > confidence that we have the functionality to run > the > > > > test. > > > > > > For > > > > > > > > > > > > > example: Can I start tasks? Can I pass > parameters? > > > Can > > > > I > > > > > > > > survive > > > > > > > > > a > > > > > > > > > > > > > context switch. Some of the tests were intended > only > > > to > > > > > > > support > > > > > > > > > > debug. > > > > > > > > > > > > > Some were created only to track down specific bugs. > > > > > > > > > > > > > > > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote: > > > > > > > > > > > > > > LTS being the Long-Term-Stable release of NuttX? > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm starting to think that it would be best to > > create > > > > new > > > > > > > suite > > > > > > > > > of > > > > > > > > > > > tests > > > > > > > > > > > > > > using Unity, using what's in OS Test as a basis > and > > > > then > > > > > > > > > expanding > > > > > > > > > > it > > > > > > > > > > > > > > further. Greg has pointed out that OS test is not > > > > > intended > > > > > > to > > > > > > > > be > > > > > > > > > a > > > > > > > > > > > > > complete > > > > > > > > > > > > > > test, so maybe a more solid testing framework > base > > > in a > > > > > new > > > > > > > > form > > > > > > > > > > > would be > > > > > > > > > > > > > > in order. At that point it would also be good to > > > > document > > > > > > and > > > > > > > > > > educate > > > > > > > > > > > > > users > > > > > > > > > > > > > > about other test options, since OS test is a > > current > > > > > > > favourite > > > > > > > > > for > > > > > > > > > > > > > showing > > > > > > > > > > > > > > PRs haven't broken anything. Maybe that's not the > > > best > > > > > > choice > > > > > > > > > > anymore > > > > > > > > > > > > > with > > > > > > > > > > > > > > this information. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't want to re-invent the wheel of what's > > already > > > > in > > > > > > > > > > `testsuite`, > > > > > > > > > > > > > but I > > > > > > > > > > > > > > think Unity might end up being lighter-weight in > > the > > > > long > > > > > > > run? > > > > > > > > > And > > > > > > > > > > > it is > > > > > > > > > > > > > > also not a Xiaomi owned/maintained framework so > it > > > may > > > > > > > receive > > > > > > > > > > better > > > > > > > > > > > > > > external support (i.e. more users rely on it?). I > > > tried > > > > > to > > > > > > > > > compile > > > > > > > > > > > the > > > > > > > > > > > > > > sim:citest configuration to run the testsuite > > example > > > > and > > > > > > see > > > > > > > > > what > > > > > > > > > > > it was > > > > > > > > > > > > > > like, but I was met with a long list of C++ > > > compilation > > > > > > > errors > > > > > > > > > that > > > > > > > > > > > I'd > > > > > > > > > > > > > > rather not deal with. There also seems to be zero > > > docs > > > > > > about > > > > > > > it > > > > > > > > > or > > > > > > > > > > > cmocka > > > > > > > > > > > > > > in our NuttX docs at the moment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > What do you think? Start a Unity test suite, add > > some > > > > > > > switches > > > > > > > > to > > > > > > > > > > > choose > > > > > > > > > > > > > > which test cases get compiled into the binary, > open > > > for > > > > > > > > > extensions > > > > > > > > > > by > > > > > > > > > > > > > users > > > > > > > > > > > > > > who know more about different subsystems? This > way > > we > > > > can > > > > > > > keep > > > > > > > > OS > > > > > > > > > > > test > > > > > > > > > > > > > how > > > > > > > > > > > > > > it is for a quick, low-resource sanity check of > > > "board > > > > > > > bringup" > > > > > > > > > > while > > > > > > > > > > > > > > having a dedicated testing application for > patches? > > > > > Someone > > > > > > > > > > changing > > > > > > > > > > > file > > > > > > > > > > > > > > system stuff can just compile to include file > > system > > > > test > > > > > > > cases > > > > > > > > > and > > > > > > > > > > > run > > > > > > > > > > > > > > that suite only as proof that their PR works. The > > > > output > > > > > is > > > > > > > > > really > > > > > > > > > > > easy > > > > > > > > > > > > > to > > > > > > > > > > > > > > glance at to determine pass/fail results. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Anyways, here's my draft demonstrating how Unity > > > works > > > > > with > > > > > > > the > > > > > > > > > FPU > > > > > > > > > > > test > > > > > > > > > > > > > > suite from OS test: > > > > > > > > > https://github.com/apache/nuttx-apps/pull/3259 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Matteo > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt < > > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >>> (..) > > > > > > > > > > > > > >>> Starting with bits and pieces of something like > > the > > > > LTS > > > > > > > might > > > > > > > > > > make > > > > > > > > > > > more > > > > > > > > > > > > > >>> sense other than its licensing problems. Since > > it > > > > > would > > > > > > > > never > > > > > > > > > be > > > > > > > > > > > > > >>> included in a distributed binary, the licensing > > > > really > > > > > > > should > > > > > > > > > not > > > > > > > > > > > > > matter > > > > > > > > > > > > > >>> much in practice. > > > > > > > > > > > > > >> Unless its GPL and some company would like to > have > > > > their > > > > > > own > > > > > > > > > > > internal > > > > > > > > > > > > > >> bits and pieces.. luckily its MIT and the whole > > > thing > > > > is > > > > > > > > > > open-source > > > > > > > > > > > > > >> anyways :-) > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Yes we had many discussions about the LTS and > this > > > > could > > > > > > be > > > > > > > > > great > > > > > > > > > > > start! > > > > > > > > > > > > > >> :-) > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> -- > > > > > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
