I like all of these ideas and would like to add: * static analysis can find simple mistakes that might have been introduced. Things like a function that forgot to return a value in some code path, or use of uninitialized variable, can be caught by static analysis.
By the way, did some recent change increase stack usage? If stacks are overflowing, you will get all kinds of weird behaviors. Maybe git bisect since a month or two ago and run some tests (like run ostest a few times for each commit being tested), see what comes up? Nathan On Sun, Apr 27, 2025 at 8:33 AM Alan C. Assis <acas...@gmail.com> wrote: > Dear NuttXers, > > In the last weeks we are seeing some degradation of NuttX reliability as > some users have reported. > > We saw it happening yesterday during our Live video: the fb command behaved > in some very strange ways: > https://www.youtube.com/watch?v=pbq3suU3g5g&t=1740s > > First it printed all the rectangles with pausing between them, then in the > next test it didn't work and after some time the board started. > > If you go back the video will notice that the "uname" command also detailed > a lot to show the results. That was not expected, NuttX is really fast. > > We have already proposed creating automated tests to help to improve NuttX, > but it alone is not enough. Some features cannot be tested easily by > automated tests. > I.E. The audio tone was broken by a commit around 2020 or early, as we only > noticed it last year when someone tried to use it. > > So these are some suggestions that we could try to help our project: > > 1) Automated Test and CI Integration (only will cover some corner cases) > It will help to detect for example if the board is not starting and if some > testings (ostest, etc) is passing. > > 2) Test Coverage Metrics > Integrate code coverage tools like gcov/lcov for unit tests, dhrystone, > coremark, etc > Display and track code coverage over time to identify untested parts of the > kernel, drivers, and libraries. > > 3) Expand and Improve Documentation > Improve Documentation/ to let end users to test boards easily. > All boards should have basic instructions explaining how to install NuttX > on it, currently almost none board has this basic instruction: i.e. > > https://nuttx.apache.org/docs/latest/platforms/arm/stm32f4/boards/stm32f4discovery/index.html > We should enhance board-specific installation guides: > How to connect the board (serial, JTAG, SWD). > How to flash NuttX (dfu-util, OpenOCD, vendor tools, etc.). > How to configure a simple project (make menuconfig, selecting board > options). > Add "Getting Started" tutorials for total beginners. > Add troubleshooting sections for common problems. > > 4) Standardize Board Port Quality > Create a checklist for each board port to ensure minimum quality: > Does ostest pass? > Do basic drivers (UART, Timer, GPIO) work? > Is SMP tested (if applicable)? > Boards that don't meet the minimum criteria are marked as "experimental" or > "unsupported". > > 5) Better Unit Testing and Mocking > Expand the apps/testing suite with more unit tests. > Use frameworks like CMocka or extend the existing ostest usage. > Mock drivers and hardware to allow kernel logic testing even without > hardware. > > 6) Stable API Guarantees > Formalize API stability between releases (similar to "Stable API" policy in > Linux kernel). > Document which APIs are considered stable and which are still experimental. > Add a deprecation process for removing/renaming public APIs. > > 7) Regression Testing > Maintain a regression test suite to ensure that previously fixed bugs do > not come back. > Basically when someone found an issue they should create a test to be > integrated into ostest to detect it in the future. > Set up automatic re-run of regression tests in CI when code is merged. > > 8) Others Performance Benchmarks Improvements > Create standard performance tests: > Boot time benchmarks > Context switch time > Interrupt latency > Track performance regressions automatically in CI. > > 9) Create Documentation/Templates to be used as reference for boards and > other common documentation > > Other idea that we could implement to validate that all most important > peripherals of all arch are working as expected: create a base board > (mainboard) with many important peripherals (sensors, audio, ethernet, usb) > and a "cartridge" board to be connected to it (we could use some existing > standard like Raspberry Pi Computer Module CM4s SODIMM: > https://datasheets.raspberrypi.com/cm4s/cm4s-datasheet.pdf or Sparkfun > MicroMod https://www.sparkfun.com/micromod). The good thing about using > MicroMod is that there are already a lot of microcontroller "cartridge" > boards that we could use. > > Please let me know what you guy think and we could plan the actions to make > it happen! > > BR, > > Alan >
