Okay, I have re-written some of the tests with cmocka and my observations are:
- cmocka has less `assert` macros than Unity to work with - The output from cmocka is a little messier to look at than Unity, perhaps that can be tweaked (?) - The output from cmocka doesn't include plain-english descriptions like "expected _, was _", but maybe this is also a good thing to cut down on strings taking up memory - cmocka test collections are much nicer to use, they allow bundling of tests into groups with per-test setup and teardown functions - cmocka test groups can be named and are displayed under a name in the output, as opposed to Unity's large blob of tests All in all, I think cmocka is a good solution too, and it honestly might be easier to manage large amounts of tests with it. I haven't done a size comparison yet, but now I am starting to think that cmocka might be better suited for this than Unity, even if it has a smaller user base. Also, while compiling for cmocka, I noticed that it includes a `cmocka` binary which looks to be a template for setting up a test-runner command-line tool. This isn't necessary for creating other cmocka test applications. So, before I move further with this, I will submit a patch to allow disabling the compilation of this cmocka binary to remove that extra bit of dependency. Matteo On Mon, Dec 22, 2025 at 11:00 PM Matteo Golin <[email protected]> wrote: > Okay this is great information! And I learned a new testing method today. > I will play with cmocka then since it seems like it could also work well > for the smaller NuttX devices. > > Thank you Xiang! > > On Mon, Dec 22, 2025, 10:57 PM Xiang Xiao <[email protected]> > wrote: > >> On Tue, Dec 23, 2025 at 10:29 AM Matteo Golin <[email protected]> >> wrote: >> >> > Thank you Xiang for the information! It makes sense of course that >> you're >> > not able to share the Xiaomi internal test suites. >> > >> > >> Actually, raiden00pl is working on upstreaming the part of the test suite >> which tests NuttX's components to the NuttX community. >> The part we can't share is that: >> >> 1. The test suite specific for openvela framework >> 2. Internal CI/CD infrastructure which is couple with our workflow and >> very different from github >> >> >> >> > Are there any modifications Xiaomi has needed to make to cmocka to >> improve >> > it's ability to run on small devices? I know some of your targets are >> > wearables which I imagine are very resource constrained? >> > >> > >> No, we don't do any size optimization for cmocka since it is already a >> small unit test framework containing one .h and one .c. >> We run many test cases(including cmocka's) on the smallest BLE only module >> which has 256KB SRAM and 2MB flash. >> >> >> >> > Can you clarify what you mean by monkey test? Is that another test >> > framework? I'm interested since you mentioned it running on watches and >> > speakers. >> > >> > >> Here is an explanation from wiki: >> https://en.wikipedia.org/wiki/Monkey_testing >> https://en.wikipedia.org/wiki/Mean_time_between_failures >> Here is the our monkey test requirement: >> >> 1. Boot 50 instances for each simulator/qemu/hardware with the product >> config >> 2. Generate the random input quickly through the fake drivers(e.g. >> touch/button/audio...) >> >> All devices need to run beyond 24 hours without crash or hang for three >> consecutive days. >> >> Thanks, >> > Matteo >> > >> > On Mon, Dec 22, 2025, 9:08 PM Xiang Xiao <[email protected]> >> > wrote: >> > >> > > On Tue, Dec 23, 2025 at 1:08 AM Matteo Golin <[email protected]> >> > > wrote: >> > > >> > > > Hello everyone, >> > > > >> > > > Thanks for all the feedback! I'm glad I discussed this here first. >> I'll >> > > try >> > > > to address everything that was brought up. >> > > > >> > > > Thanks Guiding for the input! I must just be missing some library >> on my >> > > > machine if you're able to compile it just fine. Maybe I'll track >> down >> > > that >> > > > error and see if you can recognize what's wrong. >> > > > >> > > > I don't intend to replace OS test anymore. I think it serves its >> > purpose >> > > as >> > > > a minimal sanity check. Instead, I think we need something a little >> > more >> > > > involved to test PRs since OS test can't catch many issues. It is >> > > currently >> > > > the standard everyone uses for verifying their patches and it would >> be >> > > > better to have more complete, alternate options. I do think that we >> can >> > > all >> > > > agree that OS test needs a revision of its console output so that >> it is >> > > > much easier to see pass/fail results, but that is just a matter of >> > print >> > > > statements and can be done in a different effort. >> > > > >> > > > I think the overwhelming concern here is mixing two unit testing >> > > frameworks >> > > > and no longer having one resource where we can go for tests. I share >> > > raiden >> > > > and Guiding's concern about that, which is why I'm glad I opened for >> > > > discussion here. On the one hand, Xiaomi has a pretty substantial >> > > > collection of tests in the `testsuite` application which they are >> using >> > > > internally I believe (in their CI). These run well on virtual >> machines, >> > > > which I assume is why the only configurations with that test suite >> > > included >> > > > is sim and rv-virt. I do not know nearly enough about cmocka to make >> > any >> > > > claim that it *wouldn't* work on smaller hardware devices, but it >> also >> > > > definitely looks to be less light-weight from the feature set in the >> > > > documentation. On the other hand, the Xiaomi test suites are also >> not a >> > > > complete silver bullet since they seem not to be intended to run on >> > > > hardware, and there are *a lot *of tests that need hardware >> > verification. >> > > > >> > > >> > > The test suite also runs on hardware daily, but sorry it's impossible >> to >> > > share our hardware and internal CI/CD infrastructure for the >> community. >> > > But, we already contract raiden00pl to refactor our test automation >> > > framework for public use. >> > > >> > > >> > > > In PRs we often request more than just a simulator test for certain >> > > > changes. For instance, sim isn't able to have `/dev/rand`, so that >> > needs >> > > to >> > > > be tested on a different target. >> > > > >> > > >> > > Maybe someone more familiar with Xiaomi's testing practices can weigh >> in >> > on >> > > > whether cmocka has been run on hardware. Going purely based off of >> > their >> > > > PRs, which almost exclusively cite tests on simulators, my guess is >> > that >> > > > cmocka was not designed with embedded in mind when it was written. >> > > > >> > > > >> > > CI/CD in Xiaomi is far more complex than your description: >> > > >> > > 1. Each patch will test on 10+ devices(include sim/qemu/hardware, >> > > arm/arm64/riscv/xtensa/x64, smp/up) with 1000+ test case >> > > 2. Run the full test suite(10000+ test case) every day >> > > 3. Run the monkey test every day on watch and speaker product: >> 24hour >> > X >> > > 50 instance X (sim + qemu + hardware) >> > > >> > > Many stability bugs only happen randomly in monkey test. >> > > >> > > As for size: unit testing can get large with Unity or cmocka (as >> raiden >> > > > mentioned) as more tests are added. Hence my effort to >> enable/disable >> > > > individual test suites from getting compiled. It might not be >> possible >> > to >> > > > ever get good testing ability for AVR just because the devices are >> so >> > > > small. Whatever solution is used there is probably going to involve >> > > > flashing the device multiple times with different tests. The size >> goal >> > > > should just be to target the most NuttX devices possible. >> > > > >> > > > Then there are the pros and cons of each framework (again, I don't >> know >> > > > much about cmocka so take it with a grain of salt): >> > > > >> > > > Unity's cons: >> > > > - Designed for each test-suite to be a standalone binary. This is >> > clunky >> > > on >> > > > NuttX, so `nest` is currently implemented as a mega-suite where test >> > > cases >> > > > just follow a naming convention to make it clear what is being >> tested >> > > > - setUp/tearDown can't really be leveraged because of the above >> > > > - I've noticed some funky behaviour using TEST_IGNORE in certain >> > > scenarios >> > > > but that's possibly my own configuration issue >> > > > - Would be in conflict with existing cmocka tests >> > > > >> > > > Unity's pros: >> > > > - Highly configurable, like NuttX >> > > > - Light-weight and works for embedded devices >> > > > - Significantly larger user base than cmocka >> > > > >> > > > Cmocka's cons: >> > > > - Possibly (?) not light-weight >> > > > - No documentation for existing test cases in NuttX >> > > > - Small user base >> > > > >> > > > Cmocka's pros: >> > > > - Seems to have a larger set of features, including setup/teardown >> > > > functions per-test case >> > > > - Existing tests in NuttX >> > > > >> > > > >> > > Xiaomi selected cmocka just because it has the test suite concept >> which >> > > could help us manage 10000+ tests. >> > > But we also use gtest for the c++ code base, and run many test cases >> from >> > > 3rd parties(e.g. LTP, libuv, libcxx testsuite, PSE52...) which may >> use a >> > > different test framework. >> > > So, is it hard or meaningless to stick to one unit test framework for >> an >> > > open source project. >> > > >> > > Irrespective of whether we go with Unity or cmocka, the collection of >> > tests >> > > > that we write will be intended to run on the actual hardware devices >> > > > themselves for the purposes of testing patches and doing regression >> > > tests. >> > > > As a result, the tests will be separate from the Xiaomi 'testsuite' >> > > > application which is intended for CI testing. I think this >> > realistically >> > > > nullifies choosing which framework to use on the basis of >> consistency, >> > > > since the tests are not coexisting with the cmocka tests Xiaomi has >> > > > written. Instead, the choice should be based on what best achieves >> the >> > > goal >> > > > from a functionality perspective. Unity seems like a really great >> > choice >> > > > for this in my opinion, but I am bothered by the fact that it's >> > designed >> > > > with multiple test binaries in mind. >> > > > >> > > > Let me know what you think, >> > > > Matteo >> > > > >> > > > On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <[email protected]> >> > > wrote: >> > > > >> > > > > Yes, I agree! We need data size from each solution for comparison, >> > this >> > > > is >> > > > > the first step. >> > > > > >> > > > > Maybe the conversion could be facilitated if we create equivalent >> > > > functions >> > > > > like those used in the apps/testing/testsuites: >> > > > > >> > > > > cmocka_run_group_tests -> unity_run_group_tests >> > > > > cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown >> > > > > etc... >> > > > > >> > > > > This way we avoid end up with two unit test framework, we already >> > have >> > > > the >> > > > > Makefile vs CMake precedent! :-D >> > > > > >> > > > > On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <[email protected] >> > >> > > > wrote: >> > > > > >> > > > > > The discussion about small systems doesn't make much sense >> without >> > > > > > data comparing resources of the same test cases for cmock and >> > unity. >> > > > > > >> > > > > > But let's assume that Unity is better. Should we use two >> frameworks >> > > > > > to implement unit tests for OS? Or choose one framework? If we >> > choose >> > > > > only >> > > > > > Unity, who >> > > > > > will migrate the current cmocka tests to Unity? Remember that >> cmoka >> > > > tests >> > > > > > are used in CI, so all CI tools must be also migrated. >> > > > > > Or should we use two different frameworks to test different >> things >> > > and >> > > > > make >> > > > > > even >> > > > > > more mess in apps/testing as it is now? >> > > > > > >> > > > > > No one is forcing anyone to do anything. This is an open-source >> > > > project, >> > > > > > everyone >> > > > > > should do what they feel is right. But whether the change >> should be >> > > > > merged >> > > > > > is >> > > > > > a different question. In this case, it's a very important >> question, >> > > > > because >> > > > > > unit testing >> > > > > > isn't just about test cases. It's about the entire >> infrastructure >> > > that >> > > > > > runs, captures and >> > > > > > handles test results later. Writing the test cases is the least >> of >> > > the >> > > > > > problems. >> > > > > > The infrastructure for cmoka testing is already in place, the >> > > > > > infrastructure for Unity >> > > > > > testing isn't. >> > > > > > >> > > > > > pon., 22 gru 2025 o 14:16 Alan C. Assis <[email protected]> >> > > > napisał(a): >> > > > > > >> > > > > > > Mateusz, >> > > > > > > but some chips like AVR DA have big flash, but not so much >> RAM. >> > > > > > > >> > > > > > > It is better to have some possible way to test on a small >> system, >> > > > using >> > > > > > > cmocka we are removing this option from the table. >> > > > > > > >> > > > > > > I liked the test Matteo did with Unity and I think we should >> let >> > > him >> > > > > > > continue. Forcing him to migrate to cmocka just because >> there's a >> > > > test >> > > > > > > suite from Xiaomi doesn't seem prudent to me. >> > > > > > > >> > > > > > > This is the advantage of an open mind open-source project: >> "the >> > > > > > possibility >> > > > > > > to test new ideas", if someone had said to Michal Lenc he >> > couldn't >> > > > > create >> > > > > > > NXBoot because where is already MCUBoot integrated on NuttX, >> we >> > > > didn't >> > > > > > have >> > > > > > > the option to use NXBoot today. ;-) >> > > > > > > >> > > > > > > BR, >> > > > > > > >> > > > > > > Alan >> > > > > > > >> > > > > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl < >> > [email protected]> >> > > > > > wrote: >> > > > > > > >> > > > > > > > Unit testing by itself isn't suitable for small systems. >> Ostest >> > > is >> > > > > > > > something >> > > > > > > > different (system level testing), so running it on small >> > systems >> > > > > makes >> > > > > > > > sense. >> > > > > > > > Running unit tests on small systems is pointless because you >> > > won't >> > > > > fit >> > > > > > in >> > > > > > > > FLASH >> > > > > > > > either way. You can run the same on large targets >> (preferably >> > on >> > > > > host: >> > > > > > > sim, >> > > > > > > > qemu) >> > > > > > > > and achieve the same result. >> > > > > > > > >> > > > > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <[email protected]> >> > > > > > napisał(a): >> > > > > > > > >> > > > > > > > > Is it too complicated to rewrite the cmocka tests in >> Unity? >> > > > > > > > > >> > > > > > > > > If cmoka is not feasible for small microcontrollers, like >> > AVRs >> > > > and >> > > > > > > > others, >> > > > > > > > > our test coverage will be faulty. >> > > > > > > > > >> > > > > > > > > BR, >> > > > > > > > > >> > > > > > > > > Alan >> > > > > > > > > >> > > > > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl < >> > > [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > I'm a fan of Unity and have been using it in my projects >> > for >> > > > > years, >> > > > > > > > but I >> > > > > > > > > > think >> > > > > > > > > > we should stick with one testing framework. Currently, >> > Cmocka >> > > > is >> > > > > > used >> > > > > > > > in >> > > > > > > > > > nuttx-apps, and we should stick with it. The only >> question >> > is >> > > > how >> > > > > > > well >> > > > > > > > > > Cmocka >> > > > > > > > > > is suitable for running on small systems? Will ostest >> still >> > > be >> > > > > able >> > > > > > > to >> > > > > > > > > run >> > > > > > > > > > on small MCUs? >> > > > > > > > > > With Unity this would be quite likely, so maybe an >> > exception >> > > > for >> > > > > > > Unity >> > > > > > > > > and >> > > > > > > > > > ostest would >> > > > > > > > > > be acceptable, but using it for other test cases will >> only >> > > > cause >> > > > > > > chaos. >> > > > > > > > > > >> > > > > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <[email protected] >> > >> > > > > > napisał(a): >> > > > > > > > > > >> > > > > > > > > > > Hi Matteo, >> > > > > > > > > > > >> > > > > > > > > > > Good idea ! >> > > > > > > > > > > >> > > > > > > > > > > But it is best to stick with a single testing >> framework. >> > > > > > Currently, >> > > > > > > > > many >> > > > > > > > > > > test cases are using cmocka, so I suggest reusing >> cmocka >> > > for >> > > > > > > > > consistency. >> > > > > > > > > > > >> > > > > > > > > > > ostest is a minimal basic test suite, typically used >> for >> > CT >> > > > > > > > (continuous >> > > > > > > > > > > testing) of each patch. >> > > > > > > > > > > If we move ostest to cmocka, it would make ostest >> > dependent >> > > > on >> > > > > a >> > > > > > > > > specific >> > > > > > > > > > > framework before it can be run. >> > > > > > > > > > > And It may increase the code size, which is not very >> > > friendly >> > > > > for >> > > > > > > > > > > configurations with small flash memory. >> > > > > > > > > > > >> > > > > > > > > > > As Greg mentioned: the answer depends on a tradeoff >> > between >> > > > > > effort >> > > > > > > > and >> > > > > > > > > > test >> > > > > > > > > > > coverage. >> > > > > > > > > > > >> > > > > > > > > > > We need to clarify whether we define ostest as a >> > > lightweight, >> > > > > > > > > > ready-to-run >> > > > > > > > > > > test set for quick verification, or as part of the >> full >> > > test >> > > > > > suite. >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > The follow is my compiling for cmocka: >> > > > > > > > > > > >> > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ >> > ./tools/configure.sh >> > > > > > > sim:citest >> > > > > > > > > > -j8* >> > > > > > > > > > > Copy files >> > > > > > > > > > > Select CONFIG_HOST_LINUX=y >> > > > > > > > > > > Refreshing... >> > > > > > > > > > > CP: arch/dummy/Kconfig to >> > > > > > > > > > > >> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig >> > > > > > > > > > > CP: boards/dummy/Kconfig to >> > > > > > > > > > > >> > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig >> > > > > > > > > > > LN: include/arch to arch/sim/include >> > > > > > > > > > > LN: drivers/platform to >> > > > > > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy >> > > > > > > > > > > LN: arch/sim/src/chip to >> > > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim >> > > > > > > > > > > LN: arch/sim/src/board to >> > > > > > > > > > > >> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src >> > > > > > > > > > > LN: include/arch/board to >> > > > > > > > > > > >> > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include >> > > > > > > > > > > LN: include/arch/chip to >> > > > > > > > > > > >> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim >> > > > > > > > > > > LN: platform/board to >> > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/audioutils >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/benchmarks >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/canutils >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/database >> > > > > > > > > > > mkkconfig in >> > > /home/ligd/platform/mainline/apps/examples/rust >> > > > > > > > > > > mkkconfig in >> > > > /home/ligd/platform/mainline/apps/examples/sotest >> > > > > > > > > > > mkkconfig in >> > > > /home/ligd/platform/mainline/apps/examples/module >> > > > > > > > > > > mkkconfig in >> > > > /home/ligd/platform/mainline/apps/examples/mcuboot >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/examples >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/graphics >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/industry >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/inertial >> > > > > > > > > > > mkkconfig in >> > > > > > > > /home/ligd/platform/mainline/apps/interpreters/luamodules >> > > > > > > > > > > mkkconfig in >> > /home/ligd/platform/mainline/apps/interpreters >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/mlearning >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/netutils >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee >> > > > > > > > > > > mkkconfig in >> > > > /home/ligd/platform/mainline/apps/testing/drivers >> > > > > > > > > > > mkkconfig in >> > > /home/ligd/platform/mainline/apps/testing/sched >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/testing/fs >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/testing/mm >> > > > > > > > > > > mkkconfig in >> > /home/ligd/platform/mainline/apps/testing/libc >> > > > > > > > > > > mkkconfig in >> > /home/ligd/platform/mainline/apps/testing/arch >> > > > > > > > > > > mkkconfig in >> > /home/ligd/platform/mainline/apps/testing/cxx >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/videoutils >> > > > > > > > > > > mkkconfig in >> > > > > /home/ligd/platform/mainline/apps/wireless/bluetooth >> > > > > > > > > > > mkkconfig in >> > > > > > /home/ligd/platform/mainline/apps/wireless/ieee802154 >> > > > > > > > > > > mkkconfig in >> /home/ligd/platform/mainline/apps/wireless >> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps >> > > > > > > > > > > Loaded configuration '.config' >> > > > > > > > > > > Configuration saved to '.config' >> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ >> > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make* >> > > > > > > > > > > Create version.h >> > > > > > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: >> platform/board >> > to >> > > > > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy >> > > > > > > > > > > Downloading: v1.46.0.zip >> > > > > > > > > > > ... >> > > > > > > > > > > patching file >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c >> > > > > > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 >> lines). >> > > > > > > > > > > >> > > > > > > > > > > CP: >> > > > /home/ligd/platform/mainline/nuttx/include/nuttx/config.h >> > > > > > > > > > > CP: >> > > > > /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h >> > > > > > > > > > > LD: nuttx >> > > > > > > > > > > Pac SIM with dynamic libs.. >> > > > > > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> >> > > 'sim-pac/libs/libz.so.1' >> > > > > > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> >> > > > > > > > 'sim-pac/libs/libstdc++.so.6' >> > > > > > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> >> > > 'sim-pac/libs/libc.so.6' >> > > > > > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> >> > > 'sim-pac/libs/libm.so.6' >> > > > > > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> >> > > > > > > 'sim-pac/libs/libgcc_s.so.1' >> > > > > > > > > > > '/lib64/ld-linux-x86-64.so.2' -> >> > > > 'sim-pac/ld-linux-x86-64.so.2' >> > > > > > > > > > > SIM elf with dynamic libs archive in nuttx.tgz >> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ >> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ >> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$ >> > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx * >> > > > > > > > > > > CHelloWorld: Constructor: mSecret=42 >> > > > > > > > > > > nsh: init: open failed: 2 >> > > > > > > > > > > >> > > > > > > > > > > NuttShell (NSH) NuttX-12.11.0 >> > > > > > > > > > > MOTD: username=admin password=Administrator >> > > > > > > > > > > nsh> >> > > > > > > > > > > nsh> help >> > > > > > > > > > > help usage: help [-v] [<cmd>] >> > > > > > > > > > > >> > > > > > > > > > > . df ifconfig mkfifo >> > > readlink >> > > > > > > > umount >> > > > > > > > > > > [ dmesg ifdown mkrd rm >> > > > > > > unset >> > > > > > > > > > > ? echo ifup mount >> rmdir >> > > > > > > > uptime >> > > > > > > > > > > alias env insmod mv >> rmmod >> > > > > > > > usleep >> > > > > > > > > > > unalias exec kill nslookup >> set >> > > > > > > watch >> > > > > > > > > > > basename exit pkill pidof >> sleep >> > > > > > xd >> > > > > > > > > > > break expr losetup pmconfig >> > source >> > > > > > > wait >> > > > > > > > > > > cat false ln poweroff >> test >> > > > > > > > > > > cd fdinfo ls quit >> time >> > > > > > > > > > > cp free lsmod printf >> true >> > > > > > > > > > > cmp help mkdir ps >> > > truncate >> > > > > > > > > > > dirname hexdump mkfatfs pwd >> uname >> > > > > > > > > > > >> > > > > > > > > > > Builtin Apps: >> > > > > > > > > > > cmocka >> > > > > > > > > > > cmocka_driver_block >> > > > > > > > > > > cmocka_driver_gpio >> > > > > > > > > > > cmocka_driver_pm >> > > > > > > > > > > cmocka_driver_pm_runtime >> > > > > > > > > > > cmocka_driver_regulator >> > > > > > > > > > > >> > > > > > > > > > > Matteo Golin <[email protected]> 于2025年12月22日周一 >> > > > 10:15写道： >> > > > > > > > > > > >> > > > > > > > > > > > I think so. Best to leave OS test alone and slowly >> > > replace >> > > > it >> > > > > > > with >> > > > > > > > a >> > > > > > > > > > new >> > > > > > > > > > > > tool that has improvements. The only reason I can >> see >> > to >> > > > > modify >> > > > > > > OS >> > > > > > > > > test >> > > > > > > > > > > now >> > > > > > > > > > > > would just be to have clearer output about what >> > > > > passed/failed. >> > > > > > > > > > > > >> > > > > > > > > > > > I'll come up with an improved proof of concept for a >> > new >> > > > tool >> > > > > > to >> > > > > > > > > share >> > > > > > > > > > in >> > > > > > > > > > > > the next few days. >> > > > > > > > > > > > >> > > > > > > > > > > > Matteo >> > > > > > > > > > > > >> > > > > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO < >> > > > [email protected]> >> > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > Small steps with measurable results may be the >> best >> > > > > approach >> > > > > > > > here, >> > > > > > > > > to >> > > > > > > > > > > > > create some new solution next to existing ostest >> but >> > > not >> > > > > > > > replacing >> > > > > > > > > it >> > > > > > > > > > > > > until ready? :-) >> > > > > > > > > > > > > >> > > > > > > > > > > > > -- >> > > > > > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin < >> > > > > > > > > [email protected] >> > > > > > > > > > > >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > I agree, I think it would take a long time to >> have >> > > > proper >> > > > > > > > > coverage >> > > > > > > > > > of >> > > > > > > > > > > > > NuttX >> > > > > > > > > > > > > > as a whole. Perhaps even impossible, since >> Unity is >> > > > more >> > > > > > of a >> > > > > > > > > unit >> > > > > > > > > > > > > testing >> > > > > > > > > > > > > > framework and thus there are definitely test >> cases >> > > for >> > > > > > NuttX >> > > > > > > > that >> > > > > > > > > > > would >> > > > > > > > > > > > > be >> > > > > > > > > > > > > > outside of its realm of "applicability". There >> is >> > > > still a >> > > > > > > need >> > > > > > > > > for >> > > > > > > > > > a >> > > > > > > > > > > > > > variety of testing applications and >> bench-marking >> > > > > > > applications >> > > > > > > > > for >> > > > > > > > > > > > those >> > > > > > > > > > > > > > scenarios. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > That makes sense, prerequisites would need to be >> > > > > > considered. >> > > > > > > I >> > > > > > > > > > think >> > > > > > > > > > > > > > targeting the low-hanging fruit of unit-testable >> > code >> > > > > would >> > > > > > > be >> > > > > > > > > easy >> > > > > > > > > > > in >> > > > > > > > > > > > > that >> > > > > > > > > > > > > > case, since prerequisites are mostly encoded in >> the >> > > > > Kconfig >> > > > > > > > > > > definitions >> > > > > > > > > > > > > > existing or not for features to be tested. Any >> > > complex >> > > > > > > > behaviour >> > > > > > > > > > > > > > (interrupts, multi-threading, etc.) might be >> out of >> > > the >> > > > > > realm >> > > > > > > > of >> > > > > > > > > > > what's >> > > > > > > > > > > > > > testable by Unity in the first place. We'll >> always >> > > have >> > > > > OS >> > > > > > > test >> > > > > > > > > in >> > > > > > > > > > > that >> > > > > > > > > > > > > > case, but maybe then it's better to do what Alan >> > > > > suggested >> > > > > > > and >> > > > > > > > at >> > > > > > > > > > > least >> > > > > > > > > > > > > > make its output more clear in the meantime. I >> > suppose >> > > > > RTOS >> > > > > > > > > testing >> > > > > > > > > > > is a >> > > > > > > > > > > > > > whole field in itself. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt < >> > > > > > > > > [email protected]> >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > I think that the answer depends on a tradeoff >> > > between >> > > > > > > effort >> > > > > > > > > and >> > > > > > > > > > > test >> > > > > > > > > > > > > > > coverage. I could imagine that a proper job >> > could >> > > be >> > > > > > > > man-years >> > > > > > > > > > of >> > > > > > > > > > > > > effort. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > My only concern with a re-design is to be >> certain >> > > > that >> > > > > > > tests >> > > > > > > > > are >> > > > > > > > > > > > > > > arranged so that the prerequisites of later >> test >> > > are >> > > > > > tested >> > > > > > > > > > first. >> > > > > > > > > > > > If >> > > > > > > > > > > > > > > tests are grouped by, for example, functional >> > area, >> > > > > then >> > > > > > > > > failures >> > > > > > > > > > > can >> > > > > > > > > > > > > be >> > > > > > > > > > > > > > > difficult to understand if there is a failure >> in >> > an >> > > > > > > > unverified >> > > > > > > > > > test >> > > > > > > > > > > > > > > prerequisite. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Some of the "tests" are bogus and only exist >> to >> > > > provide >> > > > > > > some >> > > > > > > > > > > minimum >> > > > > > > > > > > > > > > confidence that we have the functionality to >> run >> > > the >> > > > > > test. >> > > > > > > > For >> > > > > > > > > > > > > > > example: Can I start tasks? Can I pass >> > > parameters? >> > > > > Can >> > > > > > I >> > > > > > > > > > survive >> > > > > > > > > > > a >> > > > > > > > > > > > > > > context switch. Some of the tests were >> intended >> > > only >> > > > > to >> > > > > > > > > support >> > > > > > > > > > > > debug. >> > > > > > > > > > > > > > > Some were created only to track down specific >> > bugs. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote: >> > > > > > > > > > > > > > > > LTS being the Long-Term-Stable release of >> > NuttX? >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I'm starting to think that it would be best >> to >> > > > create >> > > > > > new >> > > > > > > > > suite >> > > > > > > > > > > of >> > > > > > > > > > > > > tests >> > > > > > > > > > > > > > > > using Unity, using what's in OS Test as a >> basis >> > > and >> > > > > > then >> > > > > > > > > > > expanding >> > > > > > > > > > > > it >> > > > > > > > > > > > > > > > further. Greg has pointed out that OS test >> is >> > not >> > > > > > > intended >> > > > > > > > to >> > > > > > > > > > be >> > > > > > > > > > > a >> > > > > > > > > > > > > > > complete >> > > > > > > > > > > > > > > > test, so maybe a more solid testing >> framework >> > > base >> > > > > in a >> > > > > > > new >> > > > > > > > > > form >> > > > > > > > > > > > > would be >> > > > > > > > > > > > > > > > in order. At that point it would also be >> good >> > to >> > > > > > document >> > > > > > > > and >> > > > > > > > > > > > educate >> > > > > > > > > > > > > > > users >> > > > > > > > > > > > > > > > about other test options, since OS test is a >> > > > current >> > > > > > > > > favourite >> > > > > > > > > > > for >> > > > > > > > > > > > > > > showing >> > > > > > > > > > > > > > > > PRs haven't broken anything. Maybe that's >> not >> > the >> > > > > best >> > > > > > > > choice >> > > > > > > > > > > > anymore >> > > > > > > > > > > > > > > with >> > > > > > > > > > > > > > > > this information. >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I don't want to re-invent the wheel of >> what's >> > > > already >> > > > > > in >> > > > > > > > > > > > `testsuite`, >> > > > > > > > > > > > > > > but I >> > > > > > > > > > > > > > > > think Unity might end up being >> lighter-weight >> > in >> > > > the >> > > > > > long >> > > > > > > > > run? >> > > > > > > > > > > And >> > > > > > > > > > > > > it is >> > > > > > > > > > > > > > > > also not a Xiaomi owned/maintained >> framework so >> > > it >> > > > > may >> > > > > > > > > receive >> > > > > > > > > > > > better >> > > > > > > > > > > > > > > > external support (i.e. more users rely on >> > it?). I >> > > > > tried >> > > > > > > to >> > > > > > > > > > > compile >> > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > sim:citest configuration to run the >> testsuite >> > > > example >> > > > > > and >> > > > > > > > see >> > > > > > > > > > > what >> > > > > > > > > > > > > it was >> > > > > > > > > > > > > > > > like, but I was met with a long list of C++ >> > > > > compilation >> > > > > > > > > errors >> > > > > > > > > > > that >> > > > > > > > > > > > > I'd >> > > > > > > > > > > > > > > > rather not deal with. There also seems to be >> > zero >> > > > > docs >> > > > > > > > about >> > > > > > > > > it >> > > > > > > > > > > or >> > > > > > > > > > > > > cmocka >> > > > > > > > > > > > > > > > in our NuttX docs at the moment. >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > What do you think? Start a Unity test suite, >> > add >> > > > some >> > > > > > > > > switches >> > > > > > > > > > to >> > > > > > > > > > > > > choose >> > > > > > > > > > > > > > > > which test cases get compiled into the >> binary, >> > > open >> > > > > for >> > > > > > > > > > > extensions >> > > > > > > > > > > > by >> > > > > > > > > > > > > > > users >> > > > > > > > > > > > > > > > who know more about different subsystems? >> This >> > > way >> > > > we >> > > > > > can >> > > > > > > > > keep >> > > > > > > > > > OS >> > > > > > > > > > > > > test >> > > > > > > > > > > > > > > how >> > > > > > > > > > > > > > > > it is for a quick, low-resource sanity >> check of >> > > > > "board >> > > > > > > > > bringup" >> > > > > > > > > > > > while >> > > > > > > > > > > > > > > > having a dedicated testing application for >> > > patches? >> > > > > > > Someone >> > > > > > > > > > > > changing >> > > > > > > > > > > > > file >> > > > > > > > > > > > > > > > system stuff can just compile to include >> file >> > > > system >> > > > > > test >> > > > > > > > > cases >> > > > > > > > > > > and >> > > > > > > > > > > > > run >> > > > > > > > > > > > > > > > that suite only as proof that their PR >> works. >> > The >> > > > > > output >> > > > > > > is >> > > > > > > > > > > really >> > > > > > > > > > > > > easy >> > > > > > > > > > > > > > > to >> > > > > > > > > > > > > > > > glance at to determine pass/fail results. >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Anyways, here's my draft demonstrating how >> > Unity >> > > > > works >> > > > > > > with >> > > > > > > > > the >> > > > > > > > > > > FPU >> > > > > > > > > > > > > test >> > > > > > > > > > > > > > > > suite from OS test: >> > > > > > > > > > > https://github.com/apache/nuttx-apps/pull/3259 >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Matteo >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO >> < >> > > > > > > > > [email protected]> >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory >> Nutt < >> > > > > > > > > > > [email protected] >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > >>> (..) >> > > > > > > > > > > > > > > >>> Starting with bits and pieces of something >> > like >> > > > the >> > > > > > LTS >> > > > > > > > > might >> > > > > > > > > > > > make >> > > > > > > > > > > > > more >> > > > > > > > > > > > > > > >>> sense other than its licensing problems. >> > Since >> > > > it >> > > > > > > would >> > > > > > > > > > never >> > > > > > > > > > > be >> > > > > > > > > > > > > > > >>> included in a distributed binary, the >> > licensing >> > > > > > really >> > > > > > > > > should >> > > > > > > > > > > not >> > > > > > > > > > > > > > > matter >> > > > > > > > > > > > > > > >>> much in practice. >> > > > > > > > > > > > > > > >> Unless its GPL and some company would like >> to >> > > have >> > > > > > their >> > > > > > > > own >> > > > > > > > > > > > > internal >> > > > > > > > > > > > > > > >> bits and pieces.. luckily its MIT and the >> > whole >> > > > > thing >> > > > > > is >> > > > > > > > > > > > open-source >> > > > > > > > > > > > > > > >> anyways :-) >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> Yes we had many discussions about the LTS >> and >> > > this >> > > > > > could >> > > > > > > > be >> > > > > > > > > > > great >> > > > > > > > > > > > > start! >> > > > > > > > > > > > > > > >> :-) >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> -- >> > > > > > > > > > > > > > > >> CeDeROM, SQ7MHZ, >> http://www.tomek.cedro.info >> > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
