> I was not aware of nuttx-testing repo. We considered NXDART (both > "tools" and "test cases") to be part of the nuttx(-rtos) repo in the > tools/nxdart location so everything stays in sync with the RTOS code > just like documentation. PRs for code, testing, and documentation > could be provided all together that way. Why do you think providing > code along with tests is bad? This seems common approach (i.e. code + > unit tests)?
The testing tool (VTFC) is a separate Python module. Ultimately, we'd like it to be available from PIP, and to achieve this, it's best to keep the entire Python package as a separate repository. Additionally, using standard Python tools like pyproject or tox is much easier when the Python module is in a separate repository. Also CI will be much simpler when this code is separated from kernel code - there is a lot of automatic tools dedicated for Python projects. The tool itself is a quite large project and I think it'll grow even larger in the future. This alone seems like a good reason to keep it in a separate repository. We can apply similar thinking to test cases, except that they won't be available as a module accessible from PIP. Automated tests are based on nuttx-apps anyway, so why should they be part of the nuttx kernel? The real dependency here is nuttx-apps, not nuttx kernel. Keeping test cases in the nuttx repo doesn't solve many problems, because any breaking changes from nuttx-apps require modifying multiple repositories anyway (apps changes in nuttx-apps and test cases changes in nuttx/tools). Moreover, changes to the nuttx repo that break tests are not welcome, as they represent breaking changes in the kernel API. This way, if a change in the kernel also requires a change in test cases, we can assume it is a breaking change and may require more attention. On the other hand, changes in nuttx-apps may affect test results and not be breaking changes, e.g., changing the strings returned by apps. So either way, the more appropriate place for test cases is nuttx-apps, not nuttx. But it still doesn't solve the other problems mentioned for Python code. Another thing is that the testing infrastructure isn't the OS; it's something separate and entirely optional (like nuttx-apps, which is a separate repository). It's not required for the kernel to function; users interested in kernel development (primarily C code) may not be interested in developing supporting tools in Python. śr., 22 paź 2025 o 14:44 Tomek CEDRO <[email protected]> napisał(a): > Hey there Raiden :-) > > Very cool! I can already test it on my local hardware NXDART setup > with plenty of different boards. Lup probably can test too when he > comes back :-) > > So far with Lup we used only shell scripts as PyTest had problems with > async and timeouts. But if you have working solution that means > probably you solved these problems. Lets see how it works :-) > > I was not aware of nuttx-testing repo. We considered NXDART (both > "tools" and "test cases") to be part of the nuttx(-rtos) repo in the > tools/nxdart location so everything stays in sync with the RTOS code > just like documentation. PRs for code, testing, and documentation > could be provided all together that way. Why do you think providing > code along with tests is bad? This seems common approach (i.e. code + > unit tests)? > > Thanks! :-) > Tomek > > > > On Wed, Oct 22, 2025 at 2:24 PM raiden00pl <[email protected]> wrote: > > > > Hi everyone, > > > > Xiaomi would like to contribute to the community an automated testing > tool > > I recently wrote, along with the dedicated NuttX test cases they use in > > their CI. > > > > The main idea is to provide a universal tool for automated testing that > can > > be > > used by the NuttX project and its users. It could be used in upstream CI > as > > well as > > in the NXDART project. It’s a similar concept to > > `nuttx/tools/ci/testrun`, but > > offers more advanced functionality. > > > > The tool is a standalone Python module that runs pytest underneath and > > extends its > > functionality with custom pytest plugins. This way, we can completely > > separate the test > > cases from the logic that executes them which is not standard pytest > usage. > > This approach provides more control over what the tool can do, but > > integrating with > > pytest requires some tricks. > > > > Test cases are written as regular pytest tests. In short, they execute > > commands > > on a NuttX target and check the returned data. Currently, SIM and QEMU > > targets > > are supported. In the future, I plan to add serial port support so that > real > > hardware testing can be done. Ideally, the tool would handle the entire > > process > > of building a NuttX image, flashing it onto hardware, and running tests > > (like Twister > > for Zephyr). > > > > The test cases are based on programs available in nuttx-apps (some tools > > aren’t > > upstream yet). Test cases from Xiaomi that don’t fit well with NuttX > > upstream > > will go into the OpenVela project. Users and companies interested in > NuttX > > will > > be able to create their own test packages and use the same tool to run > and > > manage > > them. I think this aligns quite well with the idea of distributed tests > for > > Nuttx (NXDART). > > > > The tool can generate reports and save them locally, including test > results > > and > > console logs for easier debugging. In the future, we could add options to > > compare > > various OS metrics between tests (image size, performance, etc.). > > > > The tool is already functional and can replace the current CI tool > > `nuttx/tools/ci/testrun`. > > More features will be added over time, I have a lot of ideas here. > > > > After this short introduction, I have a few questions for the community: > > > > 1. Is the community interested in adopting this new testing tool and test > > cases for > > the NuttX? Xiaomi has agreed to donate the code to the project > > > > 2. Inside Xiaomi, the project is called VTFC (Vela Testing Framework for > > Community). > > The name VTFC is a bit cryptic and refers specifically to Vela > > but Xiaomi is open > > to renaming it. Some alternative suggestions are NTFC (NuttX Testing > > Framework > > for Community) or NTF (NuttX Testing Framework). > > If anyone has better ideas, please let us know. As we all know, a cool > > project > > name is important :) > > > > 3. I think we’d need two separate repositories for this: > > > > - one for the testing tool that runs the test cases and does the rest > of > > the magic (VTFC), > > - and another for test cases dedicated to NuttX upstream. > > > > For the second one, we could use > https://github.com/apache/nuttx-testing, > > which > > is currently empty, but we’d still need one more repo. > > > > Alternatively, we could use https://github.com/Nuttx organization > which > > would make > > repository management easier (no Apache infra) but then the code will > not > > officially be under Apache and I don't know what consequences it has. > > > > Separation of the test tool and test cases from the kernel code is > > important > > because it allows for better QA for the Python code and automation. > > There will be > > quite a lot of Python coding involved, and mixing it with kernel code > > doesn’t feel good. > > > > Let me know what you think. Any suggestions are welcome :) > > > > Regards > > > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >
