Hey there Raiden :-) Very cool! I can already test it on my local hardware NXDART setup with plenty of different boards. Lup probably can test too when he comes back :-)
So far with Lup we used only shell scripts as PyTest had problems with async and timeouts. But if you have working solution that means probably you solved these problems. Lets see how it works :-) I was not aware of nuttx-testing repo. We considered NXDART (both "tools" and "test cases") to be part of the nuttx(-rtos) repo in the tools/nxdart location so everything stays in sync with the RTOS code just like documentation. PRs for code, testing, and documentation could be provided all together that way. Why do you think providing code along with tests is bad? This seems common approach (i.e. code + unit tests)? Thanks! :-) Tomek On Wed, Oct 22, 2025 at 2:24 PM raiden00pl <[email protected]> wrote: > > Hi everyone, > > Xiaomi would like to contribute to the community an automated testing tool > I recently wrote, along with the dedicated NuttX test cases they use in > their CI. > > The main idea is to provide a universal tool for automated testing that can > be > used by the NuttX project and its users. It could be used in upstream CI as > well as > in the NXDART project. It’s a similar concept to > `nuttx/tools/ci/testrun`, but > offers more advanced functionality. > > The tool is a standalone Python module that runs pytest underneath and > extends its > functionality with custom pytest plugins. This way, we can completely > separate the test > cases from the logic that executes them which is not standard pytest usage. > This approach provides more control over what the tool can do, but > integrating with > pytest requires some tricks. > > Test cases are written as regular pytest tests. In short, they execute > commands > on a NuttX target and check the returned data. Currently, SIM and QEMU > targets > are supported. In the future, I plan to add serial port support so that real > hardware testing can be done. Ideally, the tool would handle the entire > process > of building a NuttX image, flashing it onto hardware, and running tests > (like Twister > for Zephyr). > > The test cases are based on programs available in nuttx-apps (some tools > aren’t > upstream yet). Test cases from Xiaomi that don’t fit well with NuttX > upstream > will go into the OpenVela project. Users and companies interested in NuttX > will > be able to create their own test packages and use the same tool to run and > manage > them. I think this aligns quite well with the idea of distributed tests for > Nuttx (NXDART). > > The tool can generate reports and save them locally, including test results > and > console logs for easier debugging. In the future, we could add options to > compare > various OS metrics between tests (image size, performance, etc.). > > The tool is already functional and can replace the current CI tool > `nuttx/tools/ci/testrun`. > More features will be added over time, I have a lot of ideas here. > > After this short introduction, I have a few questions for the community: > > 1. Is the community interested in adopting this new testing tool and test > cases for > the NuttX? Xiaomi has agreed to donate the code to the project > > 2. Inside Xiaomi, the project is called VTFC (Vela Testing Framework for > Community). > The name VTFC is a bit cryptic and refers specifically to Vela > but Xiaomi is open > to renaming it. Some alternative suggestions are NTFC (NuttX Testing > Framework > for Community) or NTF (NuttX Testing Framework). > If anyone has better ideas, please let us know. As we all know, a cool > project > name is important :) > > 3. I think we’d need two separate repositories for this: > > - one for the testing tool that runs the test cases and does the rest of > the magic (VTFC), > - and another for test cases dedicated to NuttX upstream. > > For the second one, we could use https://github.com/apache/nuttx-testing, > which > is currently empty, but we’d still need one more repo. > > Alternatively, we could use https://github.com/Nuttx organization which > would make > repository management easier (no Apache infra) but then the code will not > officially be under Apache and I don't know what consequences it has. > > Separation of the test tool and test cases from the kernel code is > important > because it allows for better QA for the Python code and automation. > There will be > quite a lot of Python coding involved, and mixing it with kernel code > doesn’t feel good. > > Let me know what you think. Any suggestions are welcome :) > > Regards -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
