Maybe we can create a basic sanity test for each new board and or new delivered feature, in order to prove the system instability before accepting the patch. The question is: how?! I mean, I have just a few boards but maybe everyone could help. I can help to run the tests at least with the boards I have.
BR, Vinicius On Sat, Apr 26, 2025, 8:01 PM Tomek CEDRO <to...@cedro.info> wrote: > Hello Kevin! > > Thank you, that feedback is much appreciated! We all noticed quality / > reliability / self-compatibility degradation in recent months, other > users/developers reported that too, we are sorry for that, we are > working to improve things! > > There are some steps we did to prevent that happening in future: > > 1. We updated Contributing Guidelines with a strict rules and special > treatment of breaking changes that got out of hand (see > https://github.com/apache/nuttx/blob/master/CONTRIBUTING.md). > > 2. Lots of improvements were made to GitHub CI in recent months. That > should improve PR verification before it lands into master. But we are > restricted by assigned GH resources and quotas here :-( > > 3. We are working on NXDART (NuttX Distributed Automated Build and > Runtime Test Farm) that is supposed to build and runtime test on a > real world hardware boards by anyone with what they have at hand. Its > still in early stage experimenting though, free-time project, no > framework yet, and some tasks require manual interaction that blocks > full automation, additional boards, etc. We mostly use ostest for now > so not all issues may be discovered. You may want to setup your own > local build environment with the boards of interest to verify build > and runtime (i.e. ostest) of master for now and help us with testing > and ideas :-) We did some early experiments and even rPI-Zero-2W can > be used as low-cost and low-power build host! > > https://nuttx-dashboard.org/ > https://lupyuen.codeberg.page/ > https://github.com/apache/nuttx/issues/15730 > > I recently bought rPi-Pico(-W) that is RP2-B2 (RP2040?) and > sPi-Pico2(-W) RP2350. I am not familiar with the boards though. The > problem with flashing firmware that I have is requirement for manual > interaction with the board - boot button, usb attach, mount, copy > firmware file, umount, usb detach, usb attach, start runtime test. Do > you know a way how to flash these boards without manual interaction > (i.e. like with esptool or openocd) over a single USB connection so it > can be automated? Can you update make flash target so after that > firmware is flashed and ready for runtime testing? > > I have ESP32-DevKit-C and ostest passes usually fine, what else can we > add to test procedure? > > I do not have STM32H7, can you please recommend some inexpensive board > that I can buy to help with testing, and what tests beside ostest > should we perform? > > If you have some specific tests and/or requirements setting up your > own build and runtime test environment seems the best way to go - > build all commits from master for start and report us any issues > detected so we can react quickly! If you already have this kind of > setup please share some more info :-) > > You are more than welcome to join the testing and PR review process! > We really need help in this area, its only a small team of volunteers > doing the job right now :-) > > Thank you! > Tomek > > > > > > On Sat, Apr 26, 2025 at 7:10 PM Kevin Witteveen <kevinwit1...@gmail.com> > wrote: > > > > Hi NuttX, > > > > About a year ago, I started experimenting with NuttX on my RP2040 > raspberry > > pi pico board. It was a really interesting alternative to FreeRTOS or > > Zephyr. However, it wasn’t until a few months ago that I started actively > > working on NuttX. > > I made some drivers and interfaces for NuttX, however not all are > released > > yet. I also had good experience with the community and developers. > > > > However, I started to notice the reliability has gone down a lot, which > > started to block my NuttX contributions. Boards start to break here and > > there, to the point I am stuck using older NuttX versions. I’m sure this > > isn’t affecting everyone, but I might be unlucky. > > > > === Affected systems === > > > > --- RP2040 --- > > I experienced my first problems with the RP2040 recently. I am unable to > > get it to work reliably on a clean NuttX install with standard > > configurations. Sometimes the great test ‘ostest’ fails, sometimes it > > doesn't. All depending on configuration and the position of the moon. The > > bugs behave like heisenbugs. > > Sounds like memory corruption being pushed around in the kernel. > > > > --- RP2350 --- > > There is also an active issue going around the RP2350. It is even worse. > I > > recently went to test the RP2350 and run ‘ostest’. It did not get further > > than about 6 lines in my serial terminal. It did not crash dump. It just > > hangs. Huge bummer. > > > > --- ESP32-devkitc --- > > This is not a board I use, but also in recent active issues, this board > > also has similar problems. I cannot report my experience on this. > However, > > still worrying. > > > > --- STM32H7 --- > > This is a board I use at work and I am trying to introduce NuttX to this > > project. This requires USB host msc to work, which completely broke after > > NuttX version 12.3. > > I also had weird random hangs, which I am unable to replicate when I am > > observing them. Heisenbugs. > > > > === Worries === > > > > --- Developers their assumptions --- > > Recently, most issues are focussing on SMP being the problem. It might be > > entirely possible that this is the problem, but I cannot confirm. > > The part that really worries me is that every time a new PR or patch is > > made and gets merged, the issue pops up at another random location or > under > > a different behavior. > > It looks like we are just pushing the bug around until it lands on an > > unused location. > > I am having issues without SMP even. > > > > I cannot blame anyone for pushing fixes that didn't actually fix the > issue. > > I had moments where I thought I finally fixed the kernel, closing my > github > > issue, just for it to pop up again when I am not observing it. > > > > === Suggestions === > > > > I am not very familiar with the NuttX testing system. However, I suggest > > building some sort of testing farm with development boards connected to a > > system. Report which boards are actively tested. This might help catch > > problems early. Also marking boards as tested helps with my and > everyone's > > confidence using these boards. > > > > ------------------------------- > > > > I really hope these issues go away soon because NuttX is a great RTOS > and I > > actively want to use it. > > I hope this story gives some insight into the current situation from my > > perspective. > > > > Best, > > > > Kevin (Martinimarter at Github) > > > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >
