Hi Matteo,
Good idea !
But it is best to stick with a single testing framework. Currently, many
test cases are using cmocka, so I suggest reusing cmocka for consistency.
ostest is a minimal basic test suite, typically used for CT (continuous
testing) of each patch.
If we move ostest to cmocka, it would make ostest dependent on a specific
framework before it can be run.
And It may increase the code size, which is not very friendly for
configurations with small flash memory.
As Greg mentioned: the answer depends on a tradeoff between effort and test
coverage.
We need to clarify whether we define ostest as a lightweight, ready-to-run
test set for quick verification, or as part of the full test suite.
The follow is my compiling for cmocka:
*ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest -j8*
Copy files
Select CONFIG_HOST_LINUX=y
Refreshing...
CP: arch/dummy/Kconfig to
/home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
CP: boards/dummy/Kconfig to
/home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
LN: include/arch to arch/sim/include
LN: drivers/platform to /home/ligd/platform/mainline/nuttx/drivers/dummy
LN: arch/sim/src/chip to /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
LN: arch/sim/src/board to
/home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
LN: include/arch/board to
/home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
LN: include/arch/chip to
/home/ligd/platform/mainline/nuttx/arch/sim/include/sim
LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
mkkconfig in /home/ligd/platform/mainline/apps/audioutils
mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
mkkconfig in /home/ligd/platform/mainline/apps/boot
mkkconfig in /home/ligd/platform/mainline/apps/canutils
mkkconfig in /home/ligd/platform/mainline/apps/crypto
mkkconfig in /home/ligd/platform/mainline/apps/database
mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
mkkconfig in /home/ligd/platform/mainline/apps/examples/module
mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
mkkconfig in /home/ligd/platform/mainline/apps/examples
mkkconfig in /home/ligd/platform/mainline/apps/fsutils
mkkconfig in /home/ligd/platform/mainline/apps/games
mkkconfig in /home/ligd/platform/mainline/apps/graphics
mkkconfig in /home/ligd/platform/mainline/apps/industry
mkkconfig in /home/ligd/platform/mainline/apps/inertial
mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
mkkconfig in /home/ligd/platform/mainline/apps/interpreters
mkkconfig in /home/ligd/platform/mainline/apps/logging
mkkconfig in /home/ligd/platform/mainline/apps/lte
mkkconfig in /home/ligd/platform/mainline/apps/math
mkkconfig in /home/ligd/platform/mainline/apps/mlearning
mkkconfig in /home/ligd/platform/mainline/apps/netutils
mkkconfig in /home/ligd/platform/mainline/apps/sdr
mkkconfig in /home/ligd/platform/mainline/apps/system
mkkconfig in /home/ligd/platform/mainline/apps/tee
mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
mkkconfig in /home/ligd/platform/mainline/apps/testing
mkkconfig in /home/ligd/platform/mainline/apps/videoutils
mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
mkkconfig in /home/ligd/platform/mainline/apps/wireless
mkkconfig in /home/ligd/platform/mainline/apps
Loaded configuration '.config'
Configuration saved to '.config'
ligd@BAI:~/platform/mainline/nuttx$
*ligd@BAI:~/platform/mainline/nuttx$ make*
Create version.h
Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
/home/ligd/platform/mainline/apps/platform/dummy
Downloading: v1.46.0.zip
...
patching file
testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
LD: nuttx
Pac SIM with dynamic libs..
'/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
'/lib/x86_64-linux-gnu/libstdc++.so.6' -> 'sim-pac/libs/libstdc++.so.6'
'/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
'/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
'/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
'/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
SIM elf with dynamic libs archive in nuttx.tgz
ligd@BAI:~/platform/mainline/nuttx$
ligd@BAI:~/platform/mainline/nuttx$
ligd@BAI:~/platform/mainline/nuttx$
*ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
CHelloWorld: Constructor: mSecret=42
nsh: init: open failed: 2
NuttShell (NSH) NuttX-12.11.0
MOTD: username=admin password=Administrator
nsh>
nsh> help
help usage: help [-v] [<cmd>]
. df ifconfig mkfifo readlink umount
[ dmesg ifdown mkrd rm unset
? echo ifup mount rmdir uptime
alias env insmod mv rmmod usleep
unalias exec kill nslookup set watch
basename exit pkill pidof sleep xd
break expr losetup pmconfig source wait
cat false ln poweroff test
cd fdinfo ls quit time
cp free lsmod printf true
cmp help mkdir ps truncate
dirname hexdump mkfatfs pwd uname
Builtin Apps:
cmocka
cmocka_driver_block
cmocka_driver_gpio
cmocka_driver_pm
cmocka_driver_pm_runtime
cmocka_driver_regulator
Matteo Golin <[email protected]> 于2025年12月22日周一 10:15写道：
> I think so. Best to leave OS test alone and slowly replace it with a new
> tool that has improvements. The only reason I can see to modify OS test now
> would just be to have clearer output about what passed/failed.
>
> I'll come up with an improved proof of concept for a new tool to share in
> the next few days.
>
> Matteo
>
> On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <[email protected]> wrote:
>
> > Small steps with measurable results may be the best approach here, to
> > create some new solution next to existing ostest but not replacing it
> > until ready? :-)
> >
> > --
> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >
> > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <[email protected]>
> > wrote:
> > >
> > > I agree, I think it would take a long time to have proper coverage of
> > NuttX
> > > as a whole. Perhaps even impossible, since Unity is more of a unit
> > testing
> > > framework and thus there are definitely test cases for NuttX that would
> > be
> > > outside of its realm of "applicability". There is still a need for a
> > > variety of testing applications and bench-marking applications for
> those
> > > scenarios.
> > >
> > > That makes sense, prerequisites would need to be considered. I think
> > > targeting the low-hanging fruit of unit-testable code would be easy in
> > that
> > > case, since prerequisites are mostly encoded in the Kconfig definitions
> > > existing or not for features to be tested. Any complex behaviour
> > > (interrupts, multi-threading, etc.) might be out of the realm of what's
> > > testable by Unity in the first place. We'll always have OS test in that
> > > case, but maybe then it's better to do what Alan suggested and at least
> > > make its output more clear in the meantime. I suppose RTOS testing is a
> > > whole field in itself.
> > >
> > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <[email protected]>
> > wrote:
> > >
> > > > I think that the answer depends on a tradeoff between effort and test
> > > > coverage. I could imagine that a proper job could be man-years of
> > effort.
> > > >
> > > > My only concern with a re-design is to be certain that tests are
> > > > arranged so that the prerequisites of later test are tested first.
> If
> > > > tests are grouped by, for example, functional area, then failures can
> > be
> > > > difficult to understand if there is a failure in an unverified test
> > > > prerequisite.
> > > >
> > > > Some of the "tests" are bogus and only exist to provide some minimum
> > > > confidence that we have the functionality to run the test. For
> > > > example: Can I start tasks? Can I pass parameters? Can I survive a
> > > > context switch. Some of the tests were intended only to support
> debug.
> > > > Some were created only to track down specific bugs.
> > > >
> > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > >
> > > > > I'm starting to think that it would be best to create new suite of
> > tests
> > > > > using Unity, using what's in OS Test as a basis and then expanding
> it
> > > > > further. Greg has pointed out that OS test is not intended to be a
> > > > complete
> > > > > test, so maybe a more solid testing framework base in a new form
> > would be
> > > > > in order. At that point it would also be good to document and
> educate
> > > > users
> > > > > about other test options, since OS test is a current favourite for
> > > > showing
> > > > > PRs haven't broken anything. Maybe that's not the best choice
> anymore
> > > > with
> > > > > this information.
> > > > >
> > > > > I don't want to re-invent the wheel of what's already in
> `testsuite`,
> > > > but I
> > > > > think Unity might end up being lighter-weight in the long run? And
> > it is
> > > > > also not a Xiaomi owned/maintained framework so it may receive
> better
> > > > > external support (i.e. more users rely on it?). I tried to compile
> > the
> > > > > sim:citest configuration to run the testsuite example and see what
> > it was
> > > > > like, but I was met with a long list of C++ compilation errors that
> > I'd
> > > > > rather not deal with. There also seems to be zero docs about it or
> > cmocka
> > > > > in our NuttX docs at the moment.
> > > > >
> > > > > What do you think? Start a Unity test suite, add some switches to
> > choose
> > > > > which test cases get compiled into the binary, open for extensions
> by
> > > > users
> > > > > who know more about different subsystems? This way we can keep OS
> > test
> > > > how
> > > > > it is for a quick, low-resource sanity check of "board bringup"
> while
> > > > > having a dedicated testing application for patches? Someone
> changing
> > file
> > > > > system stuff can just compile to include file system test cases and
> > run
> > > > > that suite only as proof that their PR works. The output is really
> > easy
> > > > to
> > > > > glance at to determine pass/fail results.
> > > > >
> > > > > Anyways, here's my draft demonstrating how Unity works with the FPU
> > test
> > > > > suite from OS test: https://github.com/apache/nuttx-apps/pull/3259
> > > > >
> > > > > Matteo
> > > > >
> > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <[email protected]>
> > wrote:
> > > > >
> > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <[email protected]
> >
> > > > wrote:
> > > > >>> (..)
> > > > >>> Starting with bits and pieces of something like the LTS might
> make
> > more
> > > > >>> sense other than its licensing problems. Since it would never be
> > > > >>> included in a distributed binary, the licensing really should not
> > > > matter
> > > > >>> much in practice.
> > > > >> Unless its GPL and some company would like to have their own
> > internal
> > > > >> bits and pieces.. luckily its MIT and the whole thing is
> open-source
> > > > >> anyways :-)
> > > > >>
> > > > >> Yes we had many discussions about the LTS and this could be great
> > start!
> > > > >> :-)
> > > > >>
> > > > >> --
> > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > >>
> > > >
> >
>
