Hi everyone,
Xiaomi would like to contribute to the community an automated testing tool
I recently wrote, along with the dedicated NuttX test cases they use in
their CI.
The main idea is to provide a universal tool for automated testing that can
be
used by the NuttX project and its users. It could be used in upstream CI as
well as
in the NXDART project. It’s a similar concept to
`nuttx/tools/ci/testrun`, but
offers more advanced functionality.
The tool is a standalone Python module that runs pytest underneath and
extends its
functionality with custom pytest plugins. This way, we can completely
separate the test
cases from the logic that executes them which is not standard pytest usage.
This approach provides more control over what the tool can do, but
integrating with
pytest requires some tricks.
Test cases are written as regular pytest tests. In short, they execute
commands
on a NuttX target and check the returned data. Currently, SIM and QEMU
targets
are supported. In the future, I plan to add serial port support so that real
hardware testing can be done. Ideally, the tool would handle the entire
process
of building a NuttX image, flashing it onto hardware, and running tests
(like Twister
for Zephyr).
The test cases are based on programs available in nuttx-apps (some tools
aren’t
upstream yet). Test cases from Xiaomi that don’t fit well with NuttX
upstream
will go into the OpenVela project. Users and companies interested in NuttX
will
be able to create their own test packages and use the same tool to run and
manage
them. I think this aligns quite well with the idea of distributed tests for
Nuttx (NXDART).
The tool can generate reports and save them locally, including test results
and
console logs for easier debugging. In the future, we could add options to
compare
various OS metrics between tests (image size, performance, etc.).
The tool is already functional and can replace the current CI tool
`nuttx/tools/ci/testrun`.
More features will be added over time, I have a lot of ideas here.
After this short introduction, I have a few questions for the community:
1. Is the community interested in adopting this new testing tool and test
cases for
the NuttX? Xiaomi has agreed to donate the code to the project
2. Inside Xiaomi, the project is called VTFC (Vela Testing Framework for
Community).
The name VTFC is a bit cryptic and refers specifically to Vela
but Xiaomi is open
to renaming it. Some alternative suggestions are NTFC (NuttX Testing
Framework
for Community) or NTF (NuttX Testing Framework).
If anyone has better ideas, please let us know. As we all know, a cool
project
name is important :)
3. I think we’d need two separate repositories for this:
- one for the testing tool that runs the test cases and does the rest of
the magic (VTFC),
- and another for test cases dedicated to NuttX upstream.
For the second one, we could use https://github.com/apache/nuttx-testing,
which
is currently empty, but we’d still need one more repo.
Alternatively, we could use https://github.com/Nuttx organization which
would make
repository management easier (no Apache infra) but then the code will not
officially be under Apache and I don't know what consequences it has.
Separation of the test tool and test cases from the kernel code is
important
because it allows for better QA for the Python code and automation.
There will be
quite a lot of Python coding involved, and mixing it with kernel code
doesn’t feel good.
Let me know what you think. Any suggestions are welcome :)
Regards
