Hello Kevin! Thank you, that feedback is much appreciated! We all noticed quality / reliability / self-compatibility degradation in recent months, other users/developers reported that too, we are sorry for that, we are working to improve things!
There are some steps we did to prevent that happening in future: 1. We updated Contributing Guidelines with a strict rules and special treatment of breaking changes that got out of hand (see https://github.com/apache/nuttx/blob/master/CONTRIBUTING.md). 2. Lots of improvements were made to GitHub CI in recent months. That should improve PR verification before it lands into master. But we are restricted by assigned GH resources and quotas here :-( 3. We are working on NXDART (NuttX Distributed Automated Build and Runtime Test Farm) that is supposed to build and runtime test on a real world hardware boards by anyone with what they have at hand. Its still in early stage experimenting though, free-time project, no framework yet, and some tasks require manual interaction that blocks full automation, additional boards, etc. We mostly use ostest for now so not all issues may be discovered. You may want to setup your own local build environment with the boards of interest to verify build and runtime (i.e. ostest) of master for now and help us with testing and ideas :-) We did some early experiments and even rPI-Zero-2W can be used as low-cost and low-power build host! https://nuttx-dashboard.org/ https://lupyuen.codeberg.page/ https://github.com/apache/nuttx/issues/15730 I recently bought rPi-Pico(-W) that is RP2-B2 (RP2040?) and sPi-Pico2(-W) RP2350. I am not familiar with the boards though. The problem with flashing firmware that I have is requirement for manual interaction with the board - boot button, usb attach, mount, copy firmware file, umount, usb detach, usb attach, start runtime test. Do you know a way how to flash these boards without manual interaction (i.e. like with esptool or openocd) over a single USB connection so it can be automated? Can you update make flash target so after that firmware is flashed and ready for runtime testing? I have ESP32-DevKit-C and ostest passes usually fine, what else can we add to test procedure? I do not have STM32H7, can you please recommend some inexpensive board that I can buy to help with testing, and what tests beside ostest should we perform? If you have some specific tests and/or requirements setting up your own build and runtime test environment seems the best way to go - build all commits from master for start and report us any issues detected so we can react quickly! If you already have this kind of setup please share some more info :-) You are more than welcome to join the testing and PR review process! We really need help in this area, its only a small team of volunteers doing the job right now :-) Thank you! Tomek On Sat, Apr 26, 2025 at 7:10 PM Kevin Witteveen <kevinwit1...@gmail.com> wrote: > > Hi NuttX, > > About a year ago, I started experimenting with NuttX on my RP2040 raspberry > pi pico board. It was a really interesting alternative to FreeRTOS or > Zephyr. However, it wasn’t until a few months ago that I started actively > working on NuttX. > I made some drivers and interfaces for NuttX, however not all are released > yet. I also had good experience with the community and developers. > > However, I started to notice the reliability has gone down a lot, which > started to block my NuttX contributions. Boards start to break here and > there, to the point I am stuck using older NuttX versions. I’m sure this > isn’t affecting everyone, but I might be unlucky. > > === Affected systems === > > --- RP2040 --- > I experienced my first problems with the RP2040 recently. I am unable to > get it to work reliably on a clean NuttX install with standard > configurations. Sometimes the great test ‘ostest’ fails, sometimes it > doesn't. All depending on configuration and the position of the moon. The > bugs behave like heisenbugs. > Sounds like memory corruption being pushed around in the kernel. > > --- RP2350 --- > There is also an active issue going around the RP2350. It is even worse. I > recently went to test the RP2350 and run ‘ostest’. It did not get further > than about 6 lines in my serial terminal. It did not crash dump. It just > hangs. Huge bummer. > > --- ESP32-devkitc --- > This is not a board I use, but also in recent active issues, this board > also has similar problems. I cannot report my experience on this. However, > still worrying. > > --- STM32H7 --- > This is a board I use at work and I am trying to introduce NuttX to this > project. This requires USB host msc to work, which completely broke after > NuttX version 12.3. > I also had weird random hangs, which I am unable to replicate when I am > observing them. Heisenbugs. > > === Worries === > > --- Developers their assumptions --- > Recently, most issues are focussing on SMP being the problem. It might be > entirely possible that this is the problem, but I cannot confirm. > The part that really worries me is that every time a new PR or patch is > made and gets merged, the issue pops up at another random location or under > a different behavior. > It looks like we are just pushing the bug around until it lands on an > unused location. > I am having issues without SMP even. > > I cannot blame anyone for pushing fixes that didn't actually fix the issue. > I had moments where I thought I finally fixed the kernel, closing my github > issue, just for it to pop up again when I am not observing it. > > === Suggestions === > > I am not very familiar with the NuttX testing system. However, I suggest > building some sort of testing farm with development boards connected to a > system. Report which boards are actively tested. This might help catch > problems early. Also marking boards as tested helps with my and everyone's > confidence using these boards. > > ------------------------------- > > I really hope these issues go away soon because NuttX is a great RTOS and I > actively want to use it. > I hope this story gives some insight into the current situation from my > perspective. > > Best, > > Kevin (Martinimarter at Github) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
