NTFC now supports communication with the DUT via the serial port, so it's possible to use it for hardware testing. At this point the DUT reset is done using the system command provided in the YAML configuration. Not the best solution, but sufficient for the moment.
I've also added a test example for the nucleo-h743zi. By default the st-flash tool is used for DUT reset, but this can be easily changed in the configuration file to something else. pt., 7 lis 2025 o 13:35 raiden00pl <[email protected]> napisał(a): > > Right off the bat, I suggest those not familiar with Pytest to check > some examples and get a good grasp on > > fixtures, scopes and parametrization. > > That's right, knowledge of Pytest is basically the only requirement for > writing your own tests. But even without knowledge of Pytest's more > advanced features, it's possible to write tests with just simple Python. > > > How could we approach device specific functionalities? > > We have a DeviceQemu and DeviceSim, so we would need a generic > DeviceSerial to deal with UART: custom baud, stop bits, etc (not that > difficult). Then, maybe we need a vendor specific class so we can operate > properly. Take Espressif devices as example: we would need a device or > vendor specific class (DeviceEspressif) that naturally inherits the > DeviceSerial but would support entering download mode, rebooting, parsing > the initial boot logs, etc. This would allow the Product class to operate > properly when the target is from Espressif. Same could be done for other > vendors if needed, of course. > > This is where YAML configuration comes in. Basically, we can add any > customization parameter there. I think that all real hardware > with serial port communication will need similar functionality. > It would be best to avoid dependence on vendor-specific parameters > and try to generalize DUT as much as possible. > > This week I added more qemu targets (armv7a, armv7r, armv8a, riscv-64). > I also prepared the code and configuration files to enable > the use of several DTUs in test cases in the future (e.g. several > devices communicating with each other). > > wt., 4 lis 2025 o 03:43 Filipe Cavalcanti > <[email protected]> napisał(a): > >> Thanks so much for sharing this with the community, I think it is a >> massive contribution. >> >> I really liked the test cases, I think it is a good coverage and it makes >> easy to add new tests. >> >> Right off the bat, I suggest those not familiar with Pytest to check some >> examples and get a good grasp on >> fixtures, scopes and parametrization. >> >> From what I was able to study on NTFC, there is a generic template >> (DeviceCommon) which end up as DeviceQemu and DeviceSim. Those are made >> available as a product, which is actually the device under test (DUT). >> >> How could we approach device specific functionalities? >> We have a DeviceQemu and DeviceSim, so we would need a generic >> DeviceSerial to deal with UART: custom baud, stop bits, etc (not that >> difficult). Then, maybe we need a vendor specific class so we can operate >> properly. Take Espressif devices as example: we would need a device or >> vendor specific class (DeviceEspressif) that naturally inherits the >> DeviceSerial but would support entering download mode, rebooting, parsing >> the initial boot logs, etc. This would allow the Product class to operate >> properly when the target is from Espressif. Same could be done for other >> vendors if needed, of course. >> >> Filipe >> ________________________________ >> From: raiden00pl <[email protected]> >> Sent: Monday, November 3, 2025 9:41 AM >> To: [email protected] <[email protected]> >> Subject: Re: Automated Testing Framework for NuttX >> >> [External: This email originated outside Espressif] >> >> > I think it should be important to have some documentation explaining how >> to >> > add new test cases, how to add a new board (actually there is no real >> board >> > supported yet), what nuttx-nftc is and its goal.Or better yet: what it >> can >> > do and what it can't. >> >> That's right, the documentation will be gradually updated. Since I prefer >> looking at code rather than documentation, I need to gather some feedback >> on what is not clear and needs more details in doc :) >> >> > Also the "for Community" in the name passes an idea that someone just >> > created for internal use and decided to give a different version "for >> > Community", maybe it is better to define a new meaning for the C: "for >> > Compatibility", "for Completeness", etc. >> >> The name can still be changed as long as the project is in my >> repositories. >> We can even use NTF (NuttX Testing Framework), but this name is too close >> to "NFT" which can be confusing. >> >> > While I'm writing this email, I saw these FAILED messages: but not >> > indication what caused the failure: >> >> By default, all tests are executed and a more detailed log is presented >> at the end of tests. You can also look at the console log in >> `results/<test_date>/<test_case_name>.txt`. With the `--exitonfail` flag >> you can abort tests on the first failure. >> >> > >> >> external/nuttx-testing/arch/timer/test_arch_timer_integration.py::test_timerjitter_integration >> > FAILED [ 4%] >> > >> >> external/nuttx-testing/driver/framebuffer/test_driver_framebuffer_integration.py::TestDriverFramebuffer::test_driver_framebuffer_black >> > FAILED [ 4%] >> > >> >> external/nuttx-testing/driver/framebuffer/test_driver_framebuffer_integration.py::TestDriverFramebuffer::test_driver_framebuffer_white >> >> This is interesting because these tests pass on my host. You can see >> what's >> wrong in the `result/` console log. >> >> > Is there some way to improve the test speed? It is running for more than >> > 40min and still at 4%. Some tests like these that failed are taking more >> > than 7min to run, my laptop is not a slow machine: Dell XPS i7-11800H. >> So >> I >> > think on an old PC it will be even slower. Maybe it could be something >> in >> > my Ubuntu setup, did you notice sometime similar in your setup? How much >> > time does it require to finish the test on SIM? (I didn't test qemu, >> but I >> > support it will be slower) >> >> The length of test execution depends directly on the length of command >> execution on NuttX. It's impossible to speed up cases where a command >> executed on NuttX takes a long time. However, for tests executed on >> the host (sim, qemu), we can try running tests in parallel on multiple >> cores. >> This isn't currently supported and will be difficult to implement, >> but it should be possible. Unfortunately, the default pytest plugin for >> parallel >> test execution (pytest-xdist) won't work for this. >> >> Ultimately, running all tests should be infrequent (possibly once a day) >> because >> it's a costly process. For CI, test cases should be selected depending on >> the area of change in the pull request. We can use Github labels for this >> or >> add a function to select test cases depending on `git diff` passed to >> NFTC. >> >> With the `--testpath` option you can choose exactly which test to run, >> e.g. `--testpath external/nuttx-testing/arch/atomic` will only run >> arch/atomic >> test cases. >> >> For the attached SIM configuration, all test cases on my machine take >> around 1h40. >> >> > I noticed that during the test "python" is using 100% of the CPU >> (sometimes >> > it drops to 99,7%). >> >> This is normal for some test cases on SIM. Some programs running on NuttX >> take up almost 100% of the core load. >> >> >> >> niedz., 2 lis 2025 o 21:33 Alan C. Assis <[email protected]> napisał(a): >> >> > Hi Mateusz, >> > Impressive work! Kudos!!! >> > >> > I followed the steps and it compiled and tested (in fact still running >> at >> > 4% until now). >> > >> > The installation steps you wrote are easy to follow and work >> > "out-of-the-box". >> > >> > I think it should be important to have some documentation explaining >> how to >> > add new test cases, how to add a new board (actually there is no real >> board >> > supported yet), what nuttx-nftc is and its goal.Or better yet: what it >> can >> > do and what it can't. >> > >> > Also the "for Community" in the name passes an idea that someone just >> > created for internal use and decided to give a different version "for >> > Community", maybe it is better to define a new meaning for the C: "for >> > Compatibility", "for Completeness", etc. >> > >> > While I'm writing this email, I saw these FAILED messages: but not >> > indication what caused the failure: >> > >> > external/nuttx-testing/arch/space/test_space_integration.py::test_df >> PASSED >> > [ 4%] >> > >> > >> external/nuttx-testing/arch/timer/test_arch_timer_integration.py::test_timerjitter_integration >> > FAILED [ 4%] >> > >> > >> external/nuttx-testing/driver/framebuffer/test_driver_framebuffer_integration.py::TestDriverFramebuffer::test_driver_framebuffer_black >> > FAILED [ 4%] >> > >> > >> external/nuttx-testing/driver/framebuffer/test_driver_framebuffer_integration.py::TestDriverFramebuffer::test_driver_framebuffer_white >> > >> > Is there some way to improve the test speed? It is running for more than >> > 40min and still at 4%. Some tests like these that failed are taking more >> > than 7min to run, my laptop is not a slow machine: Dell XPS i7-11800H. >> So I >> > think on an old PC it will be even slower. Maybe it could be something >> in >> > my Ubuntu setup, did you notice sometime similar in your setup? How much >> > time does it require to finish the test on SIM? (I didn't test qemu, >> but I >> > support it will be slower) >> > >> > I noticed that during the test "python" is using 100% of the CPU >> (sometimes >> > it drops to 99,7%). >> > >> > These things are minor issues, I think you create something really >> powerful >> > that will help to improve the NuttX quality to some level we never saw >> > before. >> > >> > BR, >> > >> > Alan >> > >> > On Sun, Nov 2, 2025 at 2:51 PM raiden00pl <[email protected]> wrote: >> > >> > > Hi, >> > > here are the repositories if anyone would like to give them a try: >> > > >> > > - Test runner: https://github.com/szafonimateusz-mi/nuttx-ntfc >> > > - Test cases: https://github.com/szafonimateusz-mi/nuttx-testing >> > > >> > > A quick guide on how to run the tests can be found here: >> > > >> > > >> > >> https://github.com/szafonimateusz-mi/nuttx-vtfc/blob/main/docs/quickstart.rst >> > > >> > > The easiest way to get started on a Linux host is by using a >> simulator or >> > > qemu-intel64. >> > > I'll add examples for other QEMU targets later. >> > > >> > > There's still a lot of work to be done, but it's initially working and >> > > presents >> > > the general idea of the tool. Any feedback or ideas are very welcome >> :) >> > > >> > > czw., 23 paź 2025 o 13:33 Filipe Cavalcanti >> > > <[email protected]> napisał(a): >> > > >> > > > This seems very good. >> > > > >> > > > We also have a lot of testing internally on Espressif and we would >> be >> > > > willing to share them. Many of them simply cover basic use of >> > defconfigs >> > > > and some cover more complex functionality such as MCUBoot, flash >> > > > encryption, file system, etc. >> > > > >> > > > Also, we use pytest-embedded plugin for all of our tests, which >> allows >> > us >> > > > to communicate with serial devices and even QEMU. I have create the >> > > > pytest-embedded-nuttx plugin that adds support for NuttX by parsing >> the >> > > > NuttShell (either serial or QEMU). >> > > > >> > > > I think it is important we have the simulator and QEMU support since >> > > those >> > > > don't need hardware, but we should have form of adding outside >> runners >> > so >> > > > we can test on actual devices. >> > > > >> > > > Looking forward to see the Python package and the test cases! >> > > > ________________________________ >> > > > From: Nathan Hartman <[email protected]> >> > > > Sent: Wednesday, October 22, 2025 10:55 PM >> > > > To: [email protected] <[email protected]> >> > > > Subject: Re: Automated Testing Framework for NuttX >> > > > >> > > > [External: This email originated outside Espressif] >> > > > >> > > > Anything we can do to improve the testing coverage of NuttX is a >> good >> > > > thing. I am not familiar with pytest etc but conceptually the >> > description >> > > > sounds good. It also sounds like the test cases can be extended over >> > > time. >> > > > >> > > > Creating additional repositories is easy. The community just needs >> to >> > > > decide what repositories it needs and they can be created. >> > > > >> > > > Agreed we need a cool name for this! I haven't thought of a good one >> > yet >> > > > but let's solicit ideas from the community! >> > > > >> > > > Cheers, >> > > > Nathan >> > > > >> > > > On Wed, Oct 22, 2025 at 8:24 AM raiden00pl <[email protected]> >> > wrote: >> > > > >> > > > > Hi everyone, >> > > > > >> > > > > Xiaomi would like to contribute to the community an automated >> testing >> > > > tool >> > > > > I recently wrote, along with the dedicated NuttX test cases they >> use >> > in >> > > > > their CI. >> > > > > >> > > > > The main idea is to provide a universal tool for automated testing >> > that >> > > > can >> > > > > be >> > > > > used by the NuttX project and its users. It could be used in >> upstream >> > > CI >> > > > as >> > > > > well as >> > > > > in the NXDART project. It’s a similar concept to >> > > > > `nuttx/tools/ci/testrun`, but >> > > > > offers more advanced functionality. >> > > > > >> > > > > The tool is a standalone Python module that runs pytest underneath >> > and >> > > > > extends its >> > > > > functionality with custom pytest plugins. This way, we can >> completely >> > > > > separate the test >> > > > > cases from the logic that executes them which is not standard >> pytest >> > > > usage. >> > > > > This approach provides more control over what the tool can do, but >> > > > > integrating with >> > > > > pytest requires some tricks. >> > > > > >> > > > > Test cases are written as regular pytest tests. In short, they >> > execute >> > > > > commands >> > > > > on a NuttX target and check the returned data. Currently, SIM and >> > QEMU >> > > > > targets >> > > > > are supported. In the future, I plan to add serial port support so >> > that >> > > > > real >> > > > > hardware testing can be done. Ideally, the tool would handle the >> > entire >> > > > > process >> > > > > of building a NuttX image, flashing it onto hardware, and running >> > tests >> > > > > (like Twister >> > > > > for Zephyr). >> > > > > >> > > > > The test cases are based on programs available in nuttx-apps (some >> > > tools >> > > > > aren’t >> > > > > upstream yet). Test cases from Xiaomi that don’t fit well with >> NuttX >> > > > > upstream >> > > > > will go into the OpenVela project. Users and companies interested >> in >> > > > NuttX >> > > > > will >> > > > > be able to create their own test packages and use the same tool to >> > run >> > > > and >> > > > > manage >> > > > > them. I think this aligns quite well with the idea of distributed >> > tests >> > > > for >> > > > > Nuttx (NXDART). >> > > > > >> > > > > The tool can generate reports and save them locally, including >> test >> > > > results >> > > > > and >> > > > > console logs for easier debugging. In the future, we could add >> > options >> > > to >> > > > > compare >> > > > > various OS metrics between tests (image size, performance, etc.). >> > > > > >> > > > > The tool is already functional and can replace the current CI tool >> > > > > `nuttx/tools/ci/testrun`. >> > > > > More features will be added over time, I have a lot of ideas here. >> > > > > >> > > > > After this short introduction, I have a few questions for the >> > > community: >> > > > > >> > > > > 1. Is the community interested in adopting this new testing tool >> and >> > > test >> > > > > cases for >> > > > > the NuttX? Xiaomi has agreed to donate the code to the project >> > > > > >> > > > > 2. Inside Xiaomi, the project is called VTFC (Vela Testing >> Framework >> > > for >> > > > > Community). >> > > > > The name VTFC is a bit cryptic and refers specifically to Vela >> > > > > but Xiaomi is open >> > > > > to renaming it. Some alternative suggestions are NTFC (NuttX >> > Testing >> > > > > Framework >> > > > > for Community) or NTF (NuttX Testing Framework). >> > > > > If anyone has better ideas, please let us know. As we all >> know, a >> > > cool >> > > > > project >> > > > > name is important :) >> > > > > >> > > > > 3. I think we’d need two separate repositories for this: >> > > > > >> > > > > - one for the testing tool that runs the test cases and does >> the >> > > rest >> > > > of >> > > > > the magic (VTFC), >> > > > > - and another for test cases dedicated to NuttX upstream. >> > > > > >> > > > > For the second one, we could use >> > > > > https://github.com/apache/nuttx-testing, >> > > > > which >> > > > > is currently empty, but we’d still need one more repo. >> > > > > >> > > > > Alternatively, we could use https://github.com/Nuttx >> organization >> > > > which >> > > > > would make >> > > > > repository management easier (no Apache infra) but then the >> code >> > > will >> > > > > not >> > > > > officially be under Apache and I don't know what consequences >> it >> > > has. >> > > > > >> > > > > Separation of the test tool and test cases from the kernel >> code is >> > > > > important >> > > > > because it allows for better QA for the Python code and >> > automation. >> > > > > There will be >> > > > > quite a lot of Python coding involved, and mixing it with >> kernel >> > > code >> > > > > doesn’t feel good. >> > > > > >> > > > > Let me know what you think. Any suggestions are welcome :) >> > > > > >> > > > > Regards >> > > > > >> > > > >> > > >> > >> >
