I think that the answer depends on a tradeoff between effort and test
coverage. I could imagine that a proper job could be man-years of effort.
My only concern with a re-design is to be certain that tests are
arranged so that the prerequisites of later test are tested first. If
tests are grouped by, for example, functional area, then failures can be
difficult to understand if there is a failure in an unverified test
prerequisite.
Some of the "tests" are bogus and only exist to provide some minimum
confidence that we have the functionality to run the test. For
example: Can I start tasks? Can I pass parameters? Can I survive a
context switch. Some of the tests were intended only to support debug.
Some were created only to track down specific bugs.
On 12/21/25 18:39, Matteo Golin wrote:
LTS being the Long-Term-Stable release of NuttX?
I'm starting to think that it would be best to create new suite of tests
using Unity, using what's in OS Test as a basis and then expanding it
further. Greg has pointed out that OS test is not intended to be a complete
test, so maybe a more solid testing framework base in a new form would be
in order. At that point it would also be good to document and educate users
about other test options, since OS test is a current favourite for showing
PRs haven't broken anything. Maybe that's not the best choice anymore with
this information.
I don't want to re-invent the wheel of what's already in `testsuite`, but I
think Unity might end up being lighter-weight in the long run? And it is
also not a Xiaomi owned/maintained framework so it may receive better
external support (i.e. more users rely on it?). I tried to compile the
sim:citest configuration to run the testsuite example and see what it was
like, but I was met with a long list of C++ compilation errors that I'd
rather not deal with. There also seems to be zero docs about it or cmocka
in our NuttX docs at the moment.
What do you think? Start a Unity test suite, add some switches to choose
which test cases get compiled into the binary, open for extensions by users
who know more about different subsystems? This way we can keep OS test how
it is for a quick, low-resource sanity check of "board bringup" while
having a dedicated testing application for patches? Someone changing file
system stuff can just compile to include file system test cases and run
that suite only as proof that their PR works. The output is really easy to
glance at to determine pass/fail results.
Anyways, here's my draft demonstrating how Unity works with the FPU test
suite from OS test: https://github.com/apache/nuttx-apps/pull/3259
Matteo
On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <[email protected]> wrote:
On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <[email protected]> wrote:
(..)
Starting with bits and pieces of something like the LTS might make more
sense other than its licensing problems. Since it would never be
included in a distributed binary, the licensing really should not matter
much in practice.
Unless its GPL and some company would like to have their own internal
bits and pieces.. luckily its MIT and the whole thing is open-source
anyways :-)
Yes we had many discussions about the LTS and this could be great start!
:-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
