Hi, here are the repositories if anyone would like to give them a try: - Test runner: https://github.com/szafonimateusz-mi/nuttx-ntfc - Test cases: https://github.com/szafonimateusz-mi/nuttx-testing
A quick guide on how to run the tests can be found here: https://github.com/szafonimateusz-mi/nuttx-vtfc/blob/main/docs/quickstart.rst The easiest way to get started on a Linux host is by using a simulator or qemu-intel64. I'll add examples for other QEMU targets later. There's still a lot of work to be done, but it's initially working and presents the general idea of the tool. Any feedback or ideas are very welcome :) czw., 23 paź 2025 o 13:33 Filipe Cavalcanti <[email protected]> napisał(a): > This seems very good. > > We also have a lot of testing internally on Espressif and we would be > willing to share them. Many of them simply cover basic use of defconfigs > and some cover more complex functionality such as MCUBoot, flash > encryption, file system, etc. > > Also, we use pytest-embedded plugin for all of our tests, which allows us > to communicate with serial devices and even QEMU. I have create the > pytest-embedded-nuttx plugin that adds support for NuttX by parsing the > NuttShell (either serial or QEMU). > > I think it is important we have the simulator and QEMU support since those > don't need hardware, but we should have form of adding outside runners so > we can test on actual devices. > > Looking forward to see the Python package and the test cases! > ________________________________ > From: Nathan Hartman <[email protected]> > Sent: Wednesday, October 22, 2025 10:55 PM > To: [email protected] <[email protected]> > Subject: Re: Automated Testing Framework for NuttX > > [External: This email originated outside Espressif] > > Anything we can do to improve the testing coverage of NuttX is a good > thing. I am not familiar with pytest etc but conceptually the description > sounds good. It also sounds like the test cases can be extended over time. > > Creating additional repositories is easy. The community just needs to > decide what repositories it needs and they can be created. > > Agreed we need a cool name for this! I haven't thought of a good one yet > but let's solicit ideas from the community! > > Cheers, > Nathan > > On Wed, Oct 22, 2025 at 8:24 AM raiden00pl <[email protected]> wrote: > > > Hi everyone, > > > > Xiaomi would like to contribute to the community an automated testing > tool > > I recently wrote, along with the dedicated NuttX test cases they use in > > their CI. > > > > The main idea is to provide a universal tool for automated testing that > can > > be > > used by the NuttX project and its users. It could be used in upstream CI > as > > well as > > in the NXDART project. It’s a similar concept to > > `nuttx/tools/ci/testrun`, but > > offers more advanced functionality. > > > > The tool is a standalone Python module that runs pytest underneath and > > extends its > > functionality with custom pytest plugins. This way, we can completely > > separate the test > > cases from the logic that executes them which is not standard pytest > usage. > > This approach provides more control over what the tool can do, but > > integrating with > > pytest requires some tricks. > > > > Test cases are written as regular pytest tests. In short, they execute > > commands > > on a NuttX target and check the returned data. Currently, SIM and QEMU > > targets > > are supported. In the future, I plan to add serial port support so that > > real > > hardware testing can be done. Ideally, the tool would handle the entire > > process > > of building a NuttX image, flashing it onto hardware, and running tests > > (like Twister > > for Zephyr). > > > > The test cases are based on programs available in nuttx-apps (some tools > > aren’t > > upstream yet). Test cases from Xiaomi that don’t fit well with NuttX > > upstream > > will go into the OpenVela project. Users and companies interested in > NuttX > > will > > be able to create their own test packages and use the same tool to run > and > > manage > > them. I think this aligns quite well with the idea of distributed tests > for > > Nuttx (NXDART). > > > > The tool can generate reports and save them locally, including test > results > > and > > console logs for easier debugging. In the future, we could add options to > > compare > > various OS metrics between tests (image size, performance, etc.). > > > > The tool is already functional and can replace the current CI tool > > `nuttx/tools/ci/testrun`. > > More features will be added over time, I have a lot of ideas here. > > > > After this short introduction, I have a few questions for the community: > > > > 1. Is the community interested in adopting this new testing tool and test > > cases for > > the NuttX? Xiaomi has agreed to donate the code to the project > > > > 2. Inside Xiaomi, the project is called VTFC (Vela Testing Framework for > > Community). > > The name VTFC is a bit cryptic and refers specifically to Vela > > but Xiaomi is open > > to renaming it. Some alternative suggestions are NTFC (NuttX Testing > > Framework > > for Community) or NTF (NuttX Testing Framework). > > If anyone has better ideas, please let us know. As we all know, a cool > > project > > name is important :) > > > > 3. I think we’d need two separate repositories for this: > > > > - one for the testing tool that runs the test cases and does the rest > of > > the magic (VTFC), > > - and another for test cases dedicated to NuttX upstream. > > > > For the second one, we could use > > https://github.com/apache/nuttx-testing, > > which > > is currently empty, but we’d still need one more repo. > > > > Alternatively, we could use https://github.com/Nuttx organization > which > > would make > > repository management easier (no Apache infra) but then the code will > > not > > officially be under Apache and I don't know what consequences it has. > > > > Separation of the test tool and test cases from the kernel code is > > important > > because it allows for better QA for the Python code and automation. > > There will be > > quite a lot of Python coding involved, and mixing it with kernel code > > doesn’t feel good. > > > > Let me know what you think. Any suggestions are welcome :) > > > > Regards > > >
