I'm a fan of Unity and have been using it in my projects for years,
but I
think
we should stick with one testing framework. Currently, Cmocka is used
in
nuttx-apps, and we should stick with it. The only question is how well
Cmocka
is suitable for running on small systems? Will ostest still be able to
run
on small MCUs?
With Unity this would be quite likely, so maybe an exception for Unity
and
ostest would
be acceptable, but using it for other test cases will only cause
chaos.
pon., 22 gru 2025 o 10:05 Guiding Li <[email protected]> napisał(a):
> Hi Matteo,
>
> Good idea !
>
> But it is best to stick with a single testing framework. Currently, many
> test cases are using cmocka, so I suggest reusing cmocka for consistency.
>
> ostest is a minimal basic test suite, typically used for CT (continuous
> testing) of each patch.
> If we move ostest to cmocka, it would make ostest dependent on a specific
> framework before it can be run.
> And It may increase the code size, which is not very friendly for
> configurations with small flash memory.
>
> As Greg mentioned: the answer depends on a tradeoff between effort and
test
> coverage.
>
> We need to clarify whether we define ostest as a lightweight,
ready-to-run
> test set for quick verification, or as part of the full test suite.
>
>
> The follow is my compiling for cmocka:
>
> *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest
-j8*
> Copy files
> Select CONFIG_HOST_LINUX=y
> Refreshing...
> CP: arch/dummy/Kconfig to
> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> CP: boards/dummy/Kconfig to
> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> LN: include/arch to arch/sim/include
> LN: drivers/platform to /home/ligd/platform/mainline/nuttx/drivers/dummy
> LN: arch/sim/src/chip to
> /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> LN: arch/sim/src/board to
> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> LN: include/arch/board to
> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> LN: include/arch/chip to
> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
> mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> mkkconfig in /home/ligd/platform/mainline/apps/boot
> mkkconfig in /home/ligd/platform/mainline/apps/canutils
> mkkconfig in /home/ligd/platform/mainline/apps/crypto
> mkkconfig in /home/ligd/platform/mainline/apps/database
> mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> mkkconfig in /home/ligd/platform/mainline/apps/examples
> mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> mkkconfig in /home/ligd/platform/mainline/apps/games
> mkkconfig in /home/ligd/platform/mainline/apps/graphics
> mkkconfig in /home/ligd/platform/mainline/apps/industry
> mkkconfig in /home/ligd/platform/mainline/apps/inertial
> mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
> mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> mkkconfig in /home/ligd/platform/mainline/apps/logging
> mkkconfig in /home/ligd/platform/mainline/apps/lte
> mkkconfig in /home/ligd/platform/mainline/apps/math
> mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> mkkconfig in /home/ligd/platform/mainline/apps/netutils
> mkkconfig in /home/ligd/platform/mainline/apps/sdr
> mkkconfig in /home/ligd/platform/mainline/apps/system
> mkkconfig in /home/ligd/platform/mainline/apps/tee
> mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> mkkconfig in /home/ligd/platform/mainline/apps/testing
> mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
> mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
> mkkconfig in /home/ligd/platform/mainline/apps/wireless
> mkkconfig in /home/ligd/platform/mainline/apps
> Loaded configuration '.config'
> Configuration saved to '.config'
> ligd@BAI:~/platform/mainline/nuttx$
> *ligd@BAI:~/platform/mainline/nuttx$ make*
> Create version.h
> Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> /home/ligd/platform/mainline/apps/platform/dummy
> Downloading: v1.46.0.zip
> ...
> patching file
>
>
testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
>
> CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> LD: nuttx
> Pac SIM with dynamic libs..
> '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> '/lib/x86_64-linux-gnu/libstdc++.so.6' -> 'sim-pac/libs/libstdc++.so.6'
> '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
> '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> SIM elf with dynamic libs archive in nuttx.tgz
> ligd@BAI:~/platform/mainline/nuttx$
> ligd@BAI:~/platform/mainline/nuttx$
> ligd@BAI:~/platform/mainline/nuttx$
> *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> CHelloWorld: Constructor: mSecret=42
> nsh: init: open failed: 2
>
> NuttShell (NSH) NuttX-12.11.0
> MOTD: username=admin password=Administrator
> nsh>
> nsh> help
> help usage: help [-v] [<cmd>]
>
> . df ifconfig mkfifo readlink umount
> [ dmesg ifdown mkrd rm unset
> ? echo ifup mount rmdir uptime
> alias env insmod mv rmmod usleep
> unalias exec kill nslookup set watch
> basename exit pkill pidof sleep xd
> break expr losetup pmconfig source wait
> cat false ln poweroff test
> cd fdinfo ls quit time
> cp free lsmod printf true
> cmp help mkdir ps truncate
> dirname hexdump mkfatfs pwd uname
>
> Builtin Apps:
> cmocka
> cmocka_driver_block
> cmocka_driver_gpio
> cmocka_driver_pm
> cmocka_driver_pm_runtime
> cmocka_driver_regulator
>
> Matteo Golin <[email protected]> 于2025年12月22日周一 10:15写道：
>
> > I think so. Best to leave OS test alone and slowly replace it with a
new
> > tool that has improvements. The only reason I can see to modify OS test
> now
> > would just be to have clearer output about what passed/failed.
> >
> > I'll come up with an improved proof of concept for a new tool to share
in
> > the next few days.
> >
> > Matteo
> >
> > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <[email protected]> wrote:
> >
> > > Small steps with measurable results may be the best approach here, to
> > > create some new solution next to existing ostest but not replacing it
> > > until ready? :-)
> > >
> > > --
> > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > >
> > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <[email protected]
>
> > > wrote:
> > > >
> > > > I agree, I think it would take a long time to have proper coverage
of
> > > NuttX
> > > > as a whole. Perhaps even impossible, since Unity is more of a unit
> > > testing
> > > > framework and thus there are definitely test cases for NuttX that
> would
> > > be
> > > > outside of its realm of "applicability". There is still a need for
a
> > > > variety of testing applications and bench-marking applications for
> > those
> > > > scenarios.
> > > >
> > > > That makes sense, prerequisites would need to be considered. I
think
> > > > targeting the low-hanging fruit of unit-testable code would be easy
> in
> > > that
> > > > case, since prerequisites are mostly encoded in the Kconfig
> definitions
> > > > existing or not for features to be tested. Any complex behaviour
> > > > (interrupts, multi-threading, etc.) might be out of the realm of
> what's
> > > > testable by Unity in the first place. We'll always have OS test in
> that
> > > > case, but maybe then it's better to do what Alan suggested and at
> least
> > > > make its output more clear in the meantime. I suppose RTOS testing
> is a
> > > > whole field in itself.
> > > >
> > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <[email protected]>
> > > wrote:
> > > >
> > > > > I think that the answer depends on a tradeoff between effort and
> test
> > > > > coverage. I could imagine that a proper job could be man-years
of
> > > effort.
> > > > >
> > > > > My only concern with a re-design is to be certain that tests are
> > > > > arranged so that the prerequisites of later test are tested
first.
> > If
> > > > > tests are grouped by, for example, functional area, then failures
> can
> > > be
> > > > > difficult to understand if there is a failure in an unverified
test
> > > > > prerequisite.
> > > > >
> > > > > Some of the "tests" are bogus and only exist to provide some
> minimum
> > > > > confidence that we have the functionality to run the test. For
> > > > > example: Can I start tasks? Can I pass parameters? Can I
survive
> a
> > > > > context switch. Some of the tests were intended only to support
> > debug.
> > > > > Some were created only to track down specific bugs.
> > > > >
> > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > >
> > > > > > I'm starting to think that it would be best to create new suite
> of
> > > tests
> > > > > > using Unity, using what's in OS Test as a basis and then
> expanding
> > it
> > > > > > further. Greg has pointed out that OS test is not intended to
be
> a
> > > > > complete
> > > > > > test, so maybe a more solid testing framework base in a new
form
> > > would be
> > > > > > in order. At that point it would also be good to document and
> > educate
> > > > > users
> > > > > > about other test options, since OS test is a current favourite
> for
> > > > > showing
> > > > > > PRs haven't broken anything. Maybe that's not the best choice
> > anymore
> > > > > with
> > > > > > this information.
> > > > > >
> > > > > > I don't want to re-invent the wheel of what's already in
> > `testsuite`,
> > > > > but I
> > > > > > think Unity might end up being lighter-weight in the long run?
> And
> > > it is
> > > > > > also not a Xiaomi owned/maintained framework so it may receive
> > better
> > > > > > external support (i.e. more users rely on it?). I tried to
> compile
> > > the
> > > > > > sim:citest configuration to run the testsuite example and see
> what
> > > it was
> > > > > > like, but I was met with a long list of C++ compilation errors
> that
> > > I'd
> > > > > > rather not deal with. There also seems to be zero docs about it
> or
> > > cmocka
> > > > > > in our NuttX docs at the moment.
> > > > > >
> > > > > > What do you think? Start a Unity test suite, add some switches
to
> > > choose
> > > > > > which test cases get compiled into the binary, open for
> extensions
> > by
> > > > > users
> > > > > > who know more about different subsystems? This way we can keep
OS
> > > test
> > > > > how
> > > > > > it is for a quick, low-resource sanity check of "board bringup"
> > while
> > > > > > having a dedicated testing application for patches? Someone
> > changing
> > > file
> > > > > > system stuff can just compile to include file system test cases
> and
> > > run
> > > > > > that suite only as proof that their PR works. The output is
> really
> > > easy
> > > > > to
> > > > > > glance at to determine pass/fail results.
> > > > > >
> > > > > > Anyways, here's my draft demonstrating how Unity works with the
> FPU
> > > test
> > > > > > suite from OS test:
> https://github.com/apache/nuttx-apps/pull/3259
> > > > > >
> > > > > > Matteo
> > > > > >
> > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <[email protected]>
> > > wrote:
> > > > > >
> > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> [email protected]
> > >
> > > > > wrote:
> > > > > >>> (..)
> > > > > >>> Starting with bits and pieces of something like the LTS might
> > make
> > > more
> > > > > >>> sense other than its licensing problems. Since it would
never
> be
> > > > > >>> included in a distributed binary, the licensing really should
> not
> > > > > matter
> > > > > >>> much in practice.
> > > > > >> Unless its GPL and some company would like to have their own
> > > internal
> > > > > >> bits and pieces.. luckily its MIT and the whole thing is
> > open-source
> > > > > >> anyways :-)
> > > > > >>
> > > > > >> Yes we had many discussions about the LTS and this could be
> great
> > > start!
> > > > > >> :-)
> > > > > >>
> > > > > >> --
> > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > >>
> > > > >
> > >
> >
>
