One of the ideas that came to mind when working with the x86 architecture and the ACRN hypervisor was that this setup could be a powerful tool for automating complex tests for NuttX. On a multicore x86 machine with a hypervisor, we can compile different NuttX images, run them simultaneously, and test various scenarios. The output from these NuttX images can be captured on the Linux host, analyzed, and the test results reported to the user/web.
We can implement 3 types of test scenarios: 1. Self-testing NuttX images for testing various configuration options (fuzze Kconfig options with ostest etc.) 2. NuttX image that communicates with Linux host on the same machine 3. NuttX image that communicate with other NuttX images over shared memory Moreover, this can be combined with building, flashing, and running test images on external hardware connected via USB. I know this can be achieved with the ACRN hypervisor, but if any other hypervisor can run many post-launched VMs from a Linux host (Service VM in ACRN) this also should work. pt., 9 sie 2024 o 20:30 Tomek CEDRO <to...@cedro.info> napisał(a): > On Fri, Aug 9, 2024 at 3:00 PM Nathan Hartman <hartman.nat...@gmail.com> > wrote: > > I like Tomek's idea of testing Tiers, with Tier0 being "critical tests" > > that MUST pass, on all architectures and boards. > > > > I pike Alin's idea of an automatically updated test status report form, > > which can be viewed on GitHub. > > > > I recommend to list ALL supported boards in the report form, with "Need > > Test Hardware" (or something like that) shown for boards that are not > > included in any HW test cluster. The idea is to help find volunteers to > > test those boards. > > > > What happens if multiple people have the same board in their test > clusters? > > And what happens if tests pass for one person but fail for another, on > the > > same board model? We might need to be able to show: Pass, Fail, Need Test > > Hardware, or a percentage, like 2/3 (passing on 2 instances out of 3). > That > > might indicate bugs that are sensitive to timing. > > Thanks Nathan :-) I am sure we can create something useful and helpful > and I really like all those constructive "Think Before You Do"^TM > discussions here on the NuttX Dev mailing list :-) :-) > > For CI testing on the GitHub I guess we are limited to build and > simulator / qemu only. Distributed testing would be perfect so anyone > could just launch a script with some real world boards attached then > gathered results could be uploaded into a central point with total > summary tests analysis. That way we could cover almost any hardware > supported by NuttX because "someone somewhere should have that board" > :-) Also it would be prone to single point of failure (except the > central point). What a great idea! :-) > > But we should measure strengths for goals, with ENOTIME and ENOCASH > throwing all around, lets start from something simple that just works > does the job we need and allow extensions in future :-) Some design > considerations right now could make that project elegant too :-) > > Alan mentioned USB Hub - I have popular and not that expensive > (~50EUR) iTec USB 3.0 16-port 10W max with external power supply and > manual power switch at every port (PN:U3CHARGEHUB16). Did not have > time to use it yet.. and did not know that single hub ports also could > be controlled from a PC (I know FreeBSD USB stack can reset and power > off given device but that device first needs to be connected to a > port). Good hint thanks Alan :-) > > Have a good weekend folks! :-) > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >
