There are several solutions here and problems :-) 1. We require now that person that provides the patch provides a proof it is working and does not break stuff on several targets. So the patches should be well tested by the author in the first place.
2. When breaking change is detected by anyone special care is taken (i.e. discussion on the mailing list, voting, more reviewers, better testing before merge, etc). 3. Lup already has a working prototype for a tesbot that can build and run PR on a real world hardware. This uses GitHub PullRequest API. This needs to be triggered by hand after initial code review as there are security concerns of running "untrusted" external code / scripts / fetch on local network machine. 4. There are problems on boards that cannot use single USB connection to flash and runtime test (i.e. nsh or ostest). Manual interaction is required to test some boards, or additional external debug probes / uart bridges / sdcard emulators, that blocks full automation. I have two 16 port USB hubs and not all boards can be enabled all the time as boards cannot be identified. Some builds require additional external tools to be installed. etc etc. Please try to build your local build and runtime test environment for good start and use simple bash scripts for automation, any help is welcome, and ideas for improvement! :-) Thanks! :-) Tomek On Sat, Apr 26, 2025 at 9:14 PM vinicius may <hw.vinicius...@gmail.com> wrote: > Maybe we can create a basic sanity test for each new board and or new > delivered feature, in order to prove the system instability before > accepting the patch. The question is: how?! I mean, I have just a few > boards but maybe everyone could help. I can help to run the tests at least > with the boards I have. > > BR, > Vinicius > > On Sat, Apr 26, 2025, 8:01 PM Tomek CEDRO <to...@cedro.info> wrote: > > > Hello Kevin! > > > > Thank you, that feedback is much appreciated! We all noticed quality / > > reliability / self-compatibility degradation in recent months, other > > users/developers reported that too, we are sorry for that, we are > > working to improve things! > > > > There are some steps we did to prevent that happening in future: > > > > 1. We updated Contributing Guidelines with a strict rules and special > > treatment of breaking changes that got out of hand (see > > https://github.com/apache/nuttx/blob/master/CONTRIBUTING.md). > > > > 2. Lots of improvements were made to GitHub CI in recent months. That > > should improve PR verification before it lands into master. But we are > > restricted by assigned GH resources and quotas here :-( > > > > 3. We are working on NXDART (NuttX Distributed Automated Build and > > Runtime Test Farm) that is supposed to build and runtime test on a > > real world hardware boards by anyone with what they have at hand. Its > > still in early stage experimenting though, free-time project, no > > framework yet, and some tasks require manual interaction that blocks > > full automation, additional boards, etc. We mostly use ostest for now > > so not all issues may be discovered. You may want to setup your own > > local build environment with the boards of interest to verify build > > and runtime (i.e. ostest) of master for now and help us with testing > > and ideas :-) We did some early experiments and even rPI-Zero-2W can > > be used as low-cost and low-power build host! > > > > https://nuttx-dashboard.org/ > > https://lupyuen.codeberg.page/ > > https://github.com/apache/nuttx/issues/15730 > > > > I recently bought rPi-Pico(-W) that is RP2-B2 (RP2040?) and > > sPi-Pico2(-W) RP2350. I am not familiar with the boards though. The > > problem with flashing firmware that I have is requirement for manual > > interaction with the board - boot button, usb attach, mount, copy > > firmware file, umount, usb detach, usb attach, start runtime test. Do > > you know a way how to flash these boards without manual interaction > > (i.e. like with esptool or openocd) over a single USB connection so it > > can be automated? Can you update make flash target so after that > > firmware is flashed and ready for runtime testing? > > > > I have ESP32-DevKit-C and ostest passes usually fine, what else can we > > add to test procedure? > > > > I do not have STM32H7, can you please recommend some inexpensive board > > that I can buy to help with testing, and what tests beside ostest > > should we perform? > > > > If you have some specific tests and/or requirements setting up your > > own build and runtime test environment seems the best way to go - > > build all commits from master for start and report us any issues > > detected so we can react quickly! If you already have this kind of > > setup please share some more info :-) > > > > You are more than welcome to join the testing and PR review process! > > We really need help in this area, its only a small team of volunteers > > doing the job right now :-) > > > > Thank you! > > Tomek > > > > > > > > > > > > On Sat, Apr 26, 2025 at 7:10 PM Kevin Witteveen <kevinwit1...@gmail.com> > > wrote: > > > > > > Hi NuttX, > > > > > > About a year ago, I started experimenting with NuttX on my RP2040 > > raspberry > > > pi pico board. It was a really interesting alternative to FreeRTOS or > > > Zephyr. However, it wasn’t until a few months ago that I started actively > > > working on NuttX. > > > I made some drivers and interfaces for NuttX, however not all are > > released > > > yet. I also had good experience with the community and developers. > > > > > > However, I started to notice the reliability has gone down a lot, which > > > started to block my NuttX contributions. Boards start to break here and > > > there, to the point I am stuck using older NuttX versions. I’m sure this > > > isn’t affecting everyone, but I might be unlucky. > > > > > > === Affected systems === > > > > > > --- RP2040 --- > > > I experienced my first problems with the RP2040 recently. I am unable to > > > get it to work reliably on a clean NuttX install with standard > > > configurations. Sometimes the great test ‘ostest’ fails, sometimes it > > > doesn't. All depending on configuration and the position of the moon. The > > > bugs behave like heisenbugs. > > > Sounds like memory corruption being pushed around in the kernel. > > > > > > --- RP2350 --- > > > There is also an active issue going around the RP2350. It is even worse. > > I > > > recently went to test the RP2350 and run ‘ostest’. It did not get further > > > than about 6 lines in my serial terminal. It did not crash dump. It just > > > hangs. Huge bummer. > > > > > > --- ESP32-devkitc --- > > > This is not a board I use, but also in recent active issues, this board > > > also has similar problems. I cannot report my experience on this. > > However, > > > still worrying. > > > > > > --- STM32H7 --- > > > This is a board I use at work and I am trying to introduce NuttX to this > > > project. This requires USB host msc to work, which completely broke after > > > NuttX version 12.3. > > > I also had weird random hangs, which I am unable to replicate when I am > > > observing them. Heisenbugs. > > > > > > === Worries === > > > > > > --- Developers their assumptions --- > > > Recently, most issues are focussing on SMP being the problem. It might be > > > entirely possible that this is the problem, but I cannot confirm. > > > The part that really worries me is that every time a new PR or patch is > > > made and gets merged, the issue pops up at another random location or > > under > > > a different behavior. > > > It looks like we are just pushing the bug around until it lands on an > > > unused location. > > > I am having issues without SMP even. > > > > > > I cannot blame anyone for pushing fixes that didn't actually fix the > > issue. > > > I had moments where I thought I finally fixed the kernel, closing my > > github > > > issue, just for it to pop up again when I am not observing it. > > > > > > === Suggestions === > > > > > > I am not very familiar with the NuttX testing system. However, I suggest > > > building some sort of testing farm with development boards connected to a > > > system. Report which boards are actively tested. This might help catch > > > problems early. Also marking boards as tested helps with my and > > everyone's > > > confidence using these boards. > > > > > > ------------------------------- > > > > > > I really hope these issues go away soon because NuttX is a great RTOS > > and I > > > actively want to use it. > > > I hope this story gives some insight into the current situation from my > > > perspective. > > > > > > Best, > > > > > > Kevin (Martinimarter at Github) > > > > > > > > -- > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
