Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Alan C. Assis]

Hi Tomek,

Try to compile using "make -j" it will reduce the time from minutes to
seconds (but will consume more power).

BR,

Alan

On Thu, Feb 6, 2025 at 10:54 PM Tomek CEDRO  wrote:

> On Thu, Feb 6, 2025 at 7:30 PM Alan C. Assis  wrote:
> > Something to keep in mind about this distributed (build) system:
> > Not everyone has free electricity (solar panels) to run a computer server
> > 24/7. And a solution where the server owner only runs it occasionally
> won't
> > help.
> > So we need to have an option where images are built on our IC and
> > downloaded to some low-power board (e.g. Raspberry Pi) to test the user's
> > boards and send the results to our IC.
> > This way it will be more flexible and more people could contribute.
>
> Quick testing of example raspberrypi-pico:nsh build:
> 1. 3,5min: 4 core i5 CPU@2.9GHz 8GB RAM.
> 2. 19,5min: rpi-0-2w 4 core CPU@1GHz 0.5GB RAM.
>
> Did not measure power consumption yet but the overall cost seems a lot
> smaller for ~5.5x time increase on a pack of matches board size.
> rPI-0-2W is comparable to rPI-3B. For sure rPI-4 or rPI-5 will be
> faster. And this is still faster than standard PR CI build. Not bad as
> for something that could just silently work in the background :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tiago Medicci Serrano]

Hi!

I didn't say that `citest` should test "everything": of course, peripherals
can't be tested (at least most of them), but it's the most basic test for
every architecture/chip/board because it can run `ostest` and other test
applications (`smp`, for instance). IMHO, it's the most important test that
should run successfully before testing anything else. Also, it must provide
the same results on QEMU and real hardware.

Peripherals can be tested directly on real hardware based on the existing
defconfigs (I don't know if we should have a `hwtest` that tests
everything. It makes sense, but it isn't usable for the end user).

Em sex., 7 de fev. de 2025 às 08:32, raiden00pl 
escreveu:

> Support for peripherals in QEMU is very limited, and some peripherals
> supported by QEMU are not available in new hardware. So the same
> tests for qemu and real HW are just not possible for some targets.
>
> Also configurations with all peripherals/features enabled are not possible
> for many chips - we are limited by available FLASH.
>
> pt., 7 lut 2025 o 12:02 Tiago Medicci Serrano 
> napisał(a):
>
> > Hi!
> >
> > citest config: I suggest to keep citest and hardware test absolutely
> > > separate. You want to be able to change one and not the other. Not the
> > > same components will be tested.
> >
> >
> >  CI tests *include* HW testing...
> >
> > For now, we have the `citest` defconfig for some boards and they are
> > intended to be built and run on QEMU (and, by interacting with `nsh`
> > execute some test functions, like `ostest`). Extending this approach
> would
> > be the first step on HW test. If QEMU is available for some
> > arch/chip/board, it can continue being an intermediary step (we call it
> > *stage*, when applying it to CI), but as we've discussed previously, it's
> > important to test on real hardware too: test the same defconfig on both
> > QEMU and real HW (when both are available). It doesn't make sense to have
> > separate defconfigs for QEMU and HW testing. The idea of using the QEMU
> is
> > simulating real hardware, so it's expected a defconfig to work on both!
> >
> > After extending the concept of the `citest` (which already exists!) to
> test
> > them on real hardware too, we can test other existing `defconfigs` on
> real
> > hardware. For example, testing `:wifi` defconfigs connecting to an AP.
> >
> > From my experience dealing with our internal CI: *do not create
> unnecessary
> > defconfigs*.  Whenever it's required to change some configuration, do it
> > for a test case specifically by using the `kconfig-merger` and
> > `kconfig-tweak` functions when building the test case. We should only
> > provide `defconfig`s that are usable by the users. Anything else should
> be
> > avoided.
> >
> > See, we want to test everything users use when experimenting NuttX.
> > Unfortunately, users aren't that interested on `ostest` and other testing
> > applications when developing their products. That's the reason why
> `citest`
> > defconfig exists. It's an exception!
> >
> > Best regards,
> >
> > Em sex., 7 de fev. de 2025 às 05:30,  escreveu:
> >
> > > On 2025-02-06 16:02:18, Matteo Golin wrote:
> > > > I've got:
> > > > - Raspberry Pi Pico
> > > > - Raspberry Pi Pico W
> > > > - XIAO RP2040
> > > > - XIAO SAMD21
> > > > - Raspberry Pi 4B
> > > >
> > > > And would also be willing to help test, provided it's easy enough for
> > me
> > > to
> > > > automate.
> > > Just looked into my lost and forgotten drawer. As of today I have:
> > >
> > > - nucleo wl55jc1
> > > - nucleo f091rc
> > > - nucleo f303re
> > > - nucleo f429zi
> > > - stm32h735g
> > > - stm32butterfly2 (stm32f107)
> > > - mimxrt1050-evkb
> > > - mimxrt1060-evkb
> > > - mimxrt1062(teensy 4.2)
> > > - rpi pico
> > > - samc21 xplained
> > >
> > > Most of them have on board debugger. If not I also have few swd/jtag
> > > debuggers
> > > that I am not really using anymore.
> > >
> > > If you make this project so I can just put rasbian + your script to
> > init.d
> > > I can
> > > donate these, rpi4, some USB hub and some space in my lab for testing.
> > > Scripts
> > > must not be running as root. I could also create some DMZ for boards
> with
> > > networking capability to test those as well.
> > >
> > > --
> > >
> > >
> >
> .-.---.--.-.
> > > | Michal Lyszczek | Embedded C, Linux |   Company Address|  .-.
> > > opensource |
> > > | +48 727 564 419 | Software Engineer | Akacjowa 10a; 55-330 |  oo|
> > > supporter |
> > > | https://bofc.pl `.--: Brzezinka Sredzka PL | /`'\
> > > & |
> > > | GPG FF1EBFE7E3A974B1 | Bits of Code | NIP:   813 349 58 78 |(\_;/)
> > > programer |
> > >
> > >
> >
> `--^--^--^-'
> > >
> >
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[raiden00pl]

Support for peripherals in QEMU is very limited, and some peripherals
supported by QEMU are not available in new hardware. So the same
tests for qemu and real HW are just not possible for some targets.

Also configurations with all peripherals/features enabled are not possible
for many chips - we are limited by available FLASH.

pt., 7 lut 2025 o 12:02 Tiago Medicci Serrano 
napisał(a):

> Hi!
>
> citest config: I suggest to keep citest and hardware test absolutely
> > separate. You want to be able to change one and not the other. Not the
> > same components will be tested.
>
>
>  CI tests *include* HW testing...
>
> For now, we have the `citest` defconfig for some boards and they are
> intended to be built and run on QEMU (and, by interacting with `nsh`
> execute some test functions, like `ostest`). Extending this approach would
> be the first step on HW test. If QEMU is available for some
> arch/chip/board, it can continue being an intermediary step (we call it
> *stage*, when applying it to CI), but as we've discussed previously, it's
> important to test on real hardware too: test the same defconfig on both
> QEMU and real HW (when both are available). It doesn't make sense to have
> separate defconfigs for QEMU and HW testing. The idea of using the QEMU is
> simulating real hardware, so it's expected a defconfig to work on both!
>
> After extending the concept of the `citest` (which already exists!) to test
> them on real hardware too, we can test other existing `defconfigs` on real
> hardware. For example, testing `:wifi` defconfigs connecting to an AP.
>
> From my experience dealing with our internal CI: *do not create unnecessary
> defconfigs*.  Whenever it's required to change some configuration, do it
> for a test case specifically by using the `kconfig-merger` and
> `kconfig-tweak` functions when building the test case. We should only
> provide `defconfig`s that are usable by the users. Anything else should be
> avoided.
>
> See, we want to test everything users use when experimenting NuttX.
> Unfortunately, users aren't that interested on `ostest` and other testing
> applications when developing their products. That's the reason why `citest`
> defconfig exists. It's an exception!
>
> Best regards,
>
> Em sex., 7 de fev. de 2025 às 05:30,  escreveu:
>
> > On 2025-02-06 16:02:18, Matteo Golin wrote:
> > > I've got:
> > > - Raspberry Pi Pico
> > > - Raspberry Pi Pico W
> > > - XIAO RP2040
> > > - XIAO SAMD21
> > > - Raspberry Pi 4B
> > >
> > > And would also be willing to help test, provided it's easy enough for
> me
> > to
> > > automate.
> > Just looked into my lost and forgotten drawer. As of today I have:
> >
> > - nucleo wl55jc1
> > - nucleo f091rc
> > - nucleo f303re
> > - nucleo f429zi
> > - stm32h735g
> > - stm32butterfly2 (stm32f107)
> > - mimxrt1050-evkb
> > - mimxrt1060-evkb
> > - mimxrt1062(teensy 4.2)
> > - rpi pico
> > - samc21 xplained
> >
> > Most of them have on board debugger. If not I also have few swd/jtag
> > debuggers
> > that I am not really using anymore.
> >
> > If you make this project so I can just put rasbian + your script to
> init.d
> > I can
> > donate these, rpi4, some USB hub and some space in my lab for testing.
> > Scripts
> > must not be running as root. I could also create some DMZ for boards with
> > networking capability to test those as well.
> >
> > --
> >
> >
> .-.---.--.-.
> > | Michal Lyszczek | Embedded C, Linux |   Company Address|  .-.
> > opensource |
> > | +48 727 564 419 | Software Engineer | Akacjowa 10a; 55-330 |  oo|
> > supporter |
> > | https://bofc.pl `.--: Brzezinka Sredzka PL | /`'\
> > & |
> > | GPG FF1EBFE7E3A974B1 | Bits of Code | NIP:   813 349 58 78 |(\_;/)
> > programer |
> >
> >
> `--^--^--^-'
> >
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tiago Medicci Serrano]

Hi!

citest config: I suggest to keep citest and hardware test absolutely
> separate. You want to be able to change one and not the other. Not the
> same components will be tested.


 CI tests *include* HW testing...

For now, we have the `citest` defconfig for some boards and they are
intended to be built and run on QEMU (and, by interacting with `nsh`
execute some test functions, like `ostest`). Extending this approach would
be the first step on HW test. If QEMU is available for some
arch/chip/board, it can continue being an intermediary step (we call it
*stage*, when applying it to CI), but as we've discussed previously, it's
important to test on real hardware too: test the same defconfig on both
QEMU and real HW (when both are available). It doesn't make sense to have
separate defconfigs for QEMU and HW testing. The idea of using the QEMU is
simulating real hardware, so it's expected a defconfig to work on both!

After extending the concept of the `citest` (which already exists!) to test
them on real hardware too, we can test other existing `defconfigs` on real
hardware. For example, testing `:wifi` defconfigs connecting to an AP.

>From my experience dealing with our internal CI: *do not create unnecessary
defconfigs*.  Whenever it's required to change some configuration, do it
for a test case specifically by using the `kconfig-merger` and
`kconfig-tweak` functions when building the test case. We should only
provide `defconfig`s that are usable by the users. Anything else should be
avoided.

See, we want to test everything users use when experimenting NuttX.
Unfortunately, users aren't that interested on `ostest` and other testing
applications when developing their products. That's the reason why `citest`
defconfig exists. It's an exception!

Best regards,

Em sex., 7 de fev. de 2025 às 05:30,  escreveu:

> On 2025-02-06 16:02:18, Matteo Golin wrote:
> > I've got:
> > - Raspberry Pi Pico
> > - Raspberry Pi Pico W
> > - XIAO RP2040
> > - XIAO SAMD21
> > - Raspberry Pi 4B
> >
> > And would also be willing to help test, provided it's easy enough for me
> to
> > automate.
> Just looked into my lost and forgotten drawer. As of today I have:
>
> - nucleo wl55jc1
> - nucleo f091rc
> - nucleo f303re
> - nucleo f429zi
> - stm32h735g
> - stm32butterfly2 (stm32f107)
> - mimxrt1050-evkb
> - mimxrt1060-evkb
> - mimxrt1062(teensy 4.2)
> - rpi pico
> - samc21 xplained
>
> Most of them have on board debugger. If not I also have few swd/jtag
> debuggers
> that I am not really using anymore.
>
> If you make this project so I can just put rasbian + your script to init.d
> I can
> donate these, rpi4, some USB hub and some space in my lab for testing.
> Scripts
> must not be running as root. I could also create some DMZ for boards with
> networking capability to test those as well.
>
> --
>
> .-.---.--.-.
> | Michal Lyszczek | Embedded C, Linux |   Company Address|  .-.
> opensource |
> | +48 727 564 419 | Software Engineer | Akacjowa 10a; 55-330 |  oo|
> supporter |
> | https://bofc.pl `.--: Brzezinka Sredzka PL | /`'\
> & |
> | GPG FF1EBFE7E3A974B1 | Bits of Code | NIP:   813 349 58 78 |(\_;/)
> programer |
>
> `--^--^--^-'
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[michal . lyszczek]

On 2025-02-06 16:02:18, Matteo Golin wrote:
> I've got:
> - Raspberry Pi Pico
> - Raspberry Pi Pico W
> - XIAO RP2040
> - XIAO SAMD21
> - Raspberry Pi 4B
> 
> And would also be willing to help test, provided it's easy enough for me to
> automate.
Just looked into my lost and forgotten drawer. As of today I have:

- nucleo wl55jc1
- nucleo f091rc
- nucleo f303re
- nucleo f429zi
- stm32h735g
- stm32butterfly2 (stm32f107)
- mimxrt1050-evkb
- mimxrt1060-evkb
- mimxrt1062(teensy 4.2)
- rpi pico
- samc21 xplained

Most of them have on board debugger. If not I also have few swd/jtag debuggers
that I am not really using anymore.

If you make this project so I can just put rasbian + your script to init.d I can
donate these, rpi4, some USB hub and some space in my lab for testing. Scripts
must not be running as root. I could also create some DMZ for boards with
networking capability to test those as well.

-- 
.-.---.--.-.
| Michal Lyszczek | Embedded C, Linux |   Company Address|  .-. opensource |
| +48 727 564 419 | Software Engineer | Akacjowa 10a; 55-330 |  oo|  supporter |
| https://bofc.pl `.--: Brzezinka Sredzka PL | /`'\  & |
| GPG FF1EBFE7E3A974B1 | Bits of Code | NIP:   813 349 58 78 |(\_;/) programer |
`--^--^--^-'


signature.asc
Description: PGP signature

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Sebastien Lorquet]

Hi,

a bit of a summary of the 10 previous messages

NDTS: NuttX Distributed Test System

NDHT: NuttX Distributed Hardware Test

nuttx-hwtest as we already have nuttx-apps

It's not original, I know lol


citest config: I suggest to keep citest and hardware test absolutely 
separate. You want to be able to change one and not the other. Not the 
same components will be tested.


So why not name these configs hwtest? thats just an idea and I dont 
really care. just separate from citest :)



About the ability to run the tests in an intermittent manner:

This is absolutely true.

But our test system should not strongly care about that.

If it's sufficiently automated and resilient, it will just do its tasks 
in a loop as soon as it runs, and stop when the user decides to pull the 
plug or stop the script. It should be able to resume as soon as it is 
executed again.


I am not sure it is useful to build images remotely at this stage. It 
could help (maybe), but it will require a more complex architecture and 
distribution system.


So it is totally useful and interesting, but keeping it for a second 
project step looks that it could bring the testing farm up quite faster.


There is quite a lot of things to do in a first step to get a board 
built and run on a user's machine.


I'm just trying to reduce our workload so we can get the thing working 
as fast as possible. Scope creep and premature optimizations are our 
enemies.



Sebastien


On 07/02/2025 07:25, Alin Jerpelea wrote:

Hi Tomek,

thanks for taking the initiative

I can put at least
Spresense
RP 2040
RP4b


Best regards
Alin

On Fri, 7 Feb 2025, 02:54 Tomek CEDRO,  wrote:


On Thu, Feb 6, 2025 at 7:30 PM Alan C. Assis  wrote:

Something to keep in mind about this distributed (build) system:
Not everyone has free electricity (solar panels) to run a computer server
24/7. And a solution where the server owner only runs it occasionally

won't

help.
So we need to have an option where images are built on our IC and
downloaded to some low-power board (e.g. Raspberry Pi) to test the user's
boards and send the results to our IC.
This way it will be more flexible and more people could contribute.

Quick testing of example raspberrypi-pico:nsh build:
1. 3,5min: 4 core i5 CPU@2.9GHz 8GB RAM.
2. 19,5min: rpi-0-2w 4 core CPU@1GHz 0.5GB RAM.

Did not measure power consumption yet but the overall cost seems a lot
smaller for ~5.5x time increase on a pack of matches board size.
rPI-0-2W is comparable to rPI-3B. For sure rPI-4 or rPI-5 will be
faster. And this is still faster than standard PR CI build. Not bad as
for something that could just silently work in the background :-)

--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Alin Jerpelea]

Hi Tomek,

thanks for taking the initiative

I can put at least
Spresense
RP 2040
RP4b


Best regards
Alin

On Fri, 7 Feb 2025, 02:54 Tomek CEDRO,  wrote:

> On Thu, Feb 6, 2025 at 7:30 PM Alan C. Assis  wrote:
> > Something to keep in mind about this distributed (build) system:
> > Not everyone has free electricity (solar panels) to run a computer server
> > 24/7. And a solution where the server owner only runs it occasionally
> won't
> > help.
> > So we need to have an option where images are built on our IC and
> > downloaded to some low-power board (e.g. Raspberry Pi) to test the user's
> > boards and send the results to our IC.
> > This way it will be more flexible and more people could contribute.
>
> Quick testing of example raspberrypi-pico:nsh build:
> 1. 3,5min: 4 core i5 CPU@2.9GHz 8GB RAM.
> 2. 19,5min: rpi-0-2w 4 core CPU@1GHz 0.5GB RAM.
>
> Did not measure power consumption yet but the overall cost seems a lot
> smaller for ~5.5x time increase on a pack of matches board size.
> rPI-0-2W is comparable to rPI-3B. For sure rPI-4 or rPI-5 will be
> faster. And this is still faster than standard PR CI build. Not bad as
> for something that could just silently work in the background :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

On Thu, Feb 6, 2025 at 7:30 PM Alan C. Assis  wrote:
> Something to keep in mind about this distributed (build) system:
> Not everyone has free electricity (solar panels) to run a computer server
> 24/7. And a solution where the server owner only runs it occasionally won't
> help.
> So we need to have an option where images are built on our IC and
> downloaded to some low-power board (e.g. Raspberry Pi) to test the user's
> boards and send the results to our IC.
> This way it will be more flexible and more people could contribute.

Quick testing of example raspberrypi-pico:nsh build:
1. 3,5min: 4 core i5 CPU@2.9GHz 8GB RAM.
2. 19,5min: rpi-0-2w 4 core CPU@1GHz 0.5GB RAM.

Did not measure power consumption yet but the overall cost seems a lot
smaller for ~5.5x time increase on a pack of matches board size.
rPI-0-2W is comparable to rPI-3B. For sure rPI-4 or rPI-5 will be
faster. And this is still faster than standard PR CI build. Not bad as
for something that could just silently work in the background :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

On Thu, Feb 6, 2025 at 10:02 PM Matteo Golin  wrote:
> I've got:
> - Raspberry Pi Pico
> - Raspberry Pi Pico W
> - XIAO RP2040
> - XIAO SAMD21
> - Raspberry Pi 4B
> And would also be willing to help test, provided it's easy enough for me to
> automate.

Thanks Matteo! Our board base is growing :-) Yes tests should be easy
to setup and run this is the goal :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Matteo Golin]

I've got:
- Raspberry Pi Pico
- Raspberry Pi Pico W
- XIAO RP2040
- XIAO SAMD21
- Raspberry Pi 4B

And would also be willing to help test, provided it's easy enough for me to
automate.

On Thu, Feb 6, 2025 at 2:33 PM Tiago Medicci Serrano <
tiago.medi...@gmail.com> wrote:

> Hi!
>
> I wonder if that be really bad if we added miminal citest / selftes to
> > default configurations? Or this should stay minimal?
>
>
> I think one `citest` defconfig is totally different from the minimal
> configs. A minimal config is intended to be used by a user experimenting
> with NuttX and `citest` is meant to act as the reference configuration for
> testing that board.
>
> So we need to have an option where images are built on our IC and
> > downloaded to some low-power board (e.g. Raspberry Pi) to test the user's
> > boards and send the results to our IC.
>
>
> It depends on the design: I agree that the firmware should be built in a
> different "stage" and, depending on the build stage results, we can create
> a QEMU test (if not for every board, at least to test the architecture).
> Then, if successful, we can continue to the distributed hardware testing. I
> think that the first (build) and second (QEMU) stages can be made upstream.
> The last stage (HW test) is distributed. As we have successfully run two
> stages upstream, it would be safer to have it tested on our machines.
>
> Em qui., 6 de fev. de 2025 às 15:30, Alan C. Assis 
> escreveu:
>
> > I think we can ask people to suggest some names and we can select the
> best
> > name.
> >
> > Something to keep in mind about this distributed (build) system:
> >
> > Not everyone has free electricity (solar panels) to run a computer server
> > 24/7. And a solution where the server owner only runs it occasionally
> won't
> > help.
> >
> > So we need to have an option where images are built on our IC and
> > downloaded to some low-power board (e.g. Raspberry Pi) to test the user's
> > boards and send the results to our IC.
> >
> > This way it will be more flexible and more people could contribute.
> >
> > BR,
> >
> > Alan
> >
> > On Thu, Feb 6, 2025 at 3:02 PM Tomek CEDRO  wrote:
> >
> > > Folks, Alan noticed there is already project named drunx :-(
> > >
> > > https://github.com/alxolr/drunx
> > >
> > > ..and the name is not really serious :D For me its just a working
> > > slur, so if you have a better idea for name please share :-)
> > >
> > > --
> > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > >
> >
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tiago Medicci Serrano]

Hi!

I wonder if that be really bad if we added miminal citest / selftes to
> default configurations? Or this should stay minimal?


I think one `citest` defconfig is totally different from the minimal
configs. A minimal config is intended to be used by a user experimenting
with NuttX and `citest` is meant to act as the reference configuration for
testing that board.

So we need to have an option where images are built on our IC and
> downloaded to some low-power board (e.g. Raspberry Pi) to test the user's
> boards and send the results to our IC.


It depends on the design: I agree that the firmware should be built in a
different "stage" and, depending on the build stage results, we can create
a QEMU test (if not for every board, at least to test the architecture).
Then, if successful, we can continue to the distributed hardware testing. I
think that the first (build) and second (QEMU) stages can be made upstream.
The last stage (HW test) is distributed. As we have successfully run two
stages upstream, it would be safer to have it tested on our machines.

Em qui., 6 de fev. de 2025 às 15:30, Alan C. Assis 
escreveu:

> I think we can ask people to suggest some names and we can select the best
> name.
>
> Something to keep in mind about this distributed (build) system:
>
> Not everyone has free electricity (solar panels) to run a computer server
> 24/7. And a solution where the server owner only runs it occasionally won't
> help.
>
> So we need to have an option where images are built on our IC and
> downloaded to some low-power board (e.g. Raspberry Pi) to test the user's
> boards and send the results to our IC.
>
> This way it will be more flexible and more people could contribute.
>
> BR,
>
> Alan
>
> On Thu, Feb 6, 2025 at 3:02 PM Tomek CEDRO  wrote:
>
> > Folks, Alan noticed there is already project named drunx :-(
> >
> > https://github.com/alxolr/drunx
> >
> > ..and the name is not really serious :D For me its just a working
> > slur, so if you have a better idea for name please share :-)
> >
> > --
> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Alan C. Assis]

I think we can ask people to suggest some names and we can select the best
name.

Something to keep in mind about this distributed (build) system:

Not everyone has free electricity (solar panels) to run a computer server
24/7. And a solution where the server owner only runs it occasionally won't
help.

So we need to have an option where images are built on our IC and
downloaded to some low-power board (e.g. Raspberry Pi) to test the user's
boards and send the results to our IC.

This way it will be more flexible and more people could contribute.

BR,

Alan

On Thu, Feb 6, 2025 at 3:02 PM Tomek CEDRO  wrote:

> Folks, Alan noticed there is already project named drunx :-(
>
> https://github.com/alxolr/drunx
>
> ..and the name is not really serious :D For me its just a working
> slur, so if you have a better idea for name please share :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

On Thu, Feb 6, 2025 at 7:00 PM Tiago Medicci Serrano
 wrote:
> Hi!
> Boards I Have
> I have at least two units of each supported Espressif SoC that can be
> allocated for testing:
> - ESP32
> - ESP32-S2
> - ESP32-S3
> - ESP32-C3
> - ESP32-C6
> - ESP32-H2

Thanks Tiago, noted! :-)


> DRUNX Proposal (at least for the first efforts)
> That being said, I propose to start testing the `citest` defconfig (and, of
> course, turn it mandatory for every supported board). This defconfig should
> include most of the supported peripherals and run `ostest` among other
> common testing applications.

Here is the dedicated issue for Test Scenarios :-)

https://github.com/apache/nuttx/issues/15773

I wonder if that be really bad if we added miminal citest / selftes to
default configurations? Or this should stay minimal?


> Do you have any ideas on how can we make a distributed HW runner that could
> run automatically on every PR?

GitHub provides REST API for Issues and Pull Requests:

https://github.com/apache/nuttx/issues/15734

However, as Lup noted, it brings some security implications to build
unverified code at home so additional caution should be taken in this
area :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

Folks, Alan noticed there is already project named drunx :-(

https://github.com/alxolr/drunx

..and the name is not really serious :D For me its just a working
slur, so if you have a better idea for name please share :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tiago Medicci Serrano]

Hi!

Boards I Have

I have at least two units of each supported Espressif SoC that can be
allocated for testing:
- ESP32
- ESP32-S2
- ESP32-S3
- ESP32-C3
- ESP32-C6
- ESP32-H2

DRUNX Proposal (at least for the first efforts)

That being said, I propose to start testing the `citest` defconfig (and, of
course, turn it mandatory for every supported board). This defconfig should
include most of the supported peripherals and run `ostest` among other
common testing applications.

Ideally, we should try to test it on every PR (before even merging the PR,
so we should design a distributed test environment to enable it).

Espressif's Internal CI

On our internal CI, we test every `defconfig` available for our devices
(and boards). After building the firmware, we flash it and check (for every
`defconfig`) the `ps` and `ls`  commands. For specific `defconfig`s we may
run different applications. For instance, for `ostest`'s defconfig, we
always run the `ostest` application and, for Wi-Fi, we try to connect to an
AP provided by other testing device.

Do you have any ideas on how can we make a distributed HW runner that could
run automatically on every PR?

Best regards,

Em qui., 6 de fev. de 2025 às 14:18, Tomek CEDRO 
escreveu:

> On Thu, Feb 6, 2025 at 11:25 AM Sebastien Lorquet 
> wrote:
> > I can test (..)
>
> On Thu, Feb 6, 2025 at 11:40 AM Tim Hardisty 
> wrote:
> > I have: (..)
>
> Noted! TANK U =)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

On Thu, Feb 6, 2025 at 11:25 AM Sebastien Lorquet  wrote:
> I can test (..)

On Thu, Feb 6, 2025 at 11:40 AM Tim Hardisty  wrote:
> I have: (..)

Noted! TANK U =)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tim Hardisty]

I have:

 * SAMA5D2 (Arm Cortex-A) Xplained EV board
 * SAMA5D27-SOM1 EV board
 * Custom board with SAMA5D27C-1G (128Mybte SDRAM) with TFT+TS, NOR
   flash, SPI peripherals, I2C peripherals, USB, GPIO. All using NuttX
   drivers.

I need to finish the project I'm on (which is still a few months' work), 
but would then be delighted to get the EV boards back up and running if 
of interest. I rebase my custom board's code to Master regularly, so I 
get early warning of changes that break "my" things, or changes to build 
workflow, anyway.


On 06/02/2025 10:19, Tomek CEDRO wrote:

On Thu, Feb 6, 2025 at 10:01 AM raiden00pl wrote:

Testing everything as a final goal is OK, but right now it's a waste of
resources.

We are thinking ahead, planning, so anyone can attach whatever they
want / have / can port tests to, not to assume limits right from start
:-)

Yes we will start with one board, then two boards, then three, whoever
wants to add more they are free to do so it they have time and
resources :-)


I think it's much more productive to select the minimum number of boards
that give
the greatest coverage for the code.

Exactly, lets start pointing out the boards that we care about, not on
the limit of the boards :-)

I did provide list of the board that I have in the online sheet, which
one we want to run on? :-)

ARM?
* STM32*

ARM64?
* rPI*

RISC-V?
* ESP32-C3.
* Sopgo.

XTENSA?
* ESP32*

???

:-)



It seems it is already spread over many various emails and github

issues. I think emails are good for discussions, while github is good
for noting conclusions down. But if you like Confluence we may try it
why not :-)

This is exactly the problem with email. This is not the first time these
topics are
discussed. Similar discussions and ideas have already been here. Where are
they now? Somewhere in dozens of emails over the years that I can't even
find.
Email for discussion and decision-making - OK, email as a knowledge base
and
an organized place with content - not really.

Lets just make small prototypes, small measurable steps.

GitHub for bullet point notes / hints / tasks / conclusions :-)

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

On Thu, Feb 6, 2025 at 10:01 AM raiden00pl  wrote:
> Testing everything as a final goal is OK, but right now it's a waste of
> resources.

We are thinking ahead, planning, so anyone can attach whatever they
want / have / can port tests to, not to assume limits right from start
:-)

Yes we will start with one board, then two boards, then three, whoever
wants to add more they are free to do so it they have time and
resources :-)

> I think it's much more productive to select the minimum number of boards
> that give
> the greatest coverage for the code.

Exactly, lets start pointing out the boards that we care about, not on
the limit of the boards :-)

I did provide list of the board that I have in the online sheet, which
one we want to run on? :-)

ARM?
* STM32*

ARM64?
* rPI*

RISC-V?
* ESP32-C3.
* Sopgo.

XTENSA?
* ESP32*

???

:-)


> > It seems it is already spread over many various emails and github
> issues. I think emails are good for discussions, while github is good
> for noting conclusions down. But if you like Confluence we may try it
> why not :-)
>
> This is exactly the problem with email. This is not the first time these
> topics are
> discussed. Similar discussions and ideas have already been here. Where are
> they now? Somewhere in dozens of emails over the years that I can't even
> find.
> Email for discussion and decision-making - OK, email as a knowledge base
> and
> an organized place with content - not really.

Lets just make small prototypes, small measurable steps.

GitHub for bullet point notes / hints / tasks / conclusions :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Sebastien Lorquet]

I can test

-nucleo-H743ZI (MB1137 B-01)

-nucleo-H743ZI2

-STM32F492I-disco (MB1075 B-01 it has peripherals and external RAM)

-STM32L1 discovery (MP963 C)

-ESP32 devkit

-raspberry pi pico

-raspberry pi pico 2

-our custom boards using STM32F429, STM32H743ZI

I will consider acquiring more boards as testing need arise.

Sebastien


On 06/02/2025 11:19, Tomek CEDRO wrote:

On Thu, Feb 6, 2025 at 10:01 AM raiden00pl  wrote:

Testing everything as a final goal is OK, but right now it's a waste of
resources.

We are thinking ahead, planning, so anyone can attach whatever they
want / have / can port tests to, not to assume limits right from start
:-)

Yes we will start with one board, then two boards, then three, whoever
wants to add more they are free to do so it they have time and
resources :-)


I think it's much more productive to select the minimum number of boards
that give
the greatest coverage for the code.

Exactly, lets start pointing out the boards that we care about, not on
the limit of the boards :-)

I did provide list of the board that I have in the online sheet, which
one we want to run on? :-)

ARM?
* STM32*

ARM64?
* rPI*

RISC-V?
* ESP32-C3.
* Sopgo.

XTENSA?
* ESP32*

???

:-)



It seems it is already spread over many various emails and github

issues. I think emails are good for discussions, while github is good
for noting conclusions down. But if you like Confluence we may try it
why not :-)

This is exactly the problem with email. This is not the first time these
topics are
discussed. Similar discussions and ideas have already been here. Where are
they now? Somewhere in dozens of emails over the years that I can't even
find.
Email for discussion and decision-making - OK, email as a knowledge base
and
an organized place with content - not really.

Lets just make small prototypes, small measurable steps.

GitHub for bullet point notes / hints / tasks / conclusions :-)

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Sebastien Lorquet]

Hi,

On 06/02/2025 10:00, raiden00pl wrote:

Chips with the same architecture use the same code,
so when we test a more advanced chip, we also test the code for a less
advanced chip.


Same code on slightly different chips is interesting because it 
underpins the least documented differences between these arches.


Of couse board choice should be be made wisely and must be adapted to 
the number of participants.


Again, I think there is no minimum. even testing on a single random 
board once a week will be useful.




The flaw in your reasoning is that we don't have many people around the
world,
just a few, and even fewer, who are willing to get involved in this project
🙂
Another thing is that some of the boards are obsolete and impossible to get.


Dont underestimate the depth of many dusty drawers :)

If we advertise this project it may get unexpected contributions.

If deployment is as easy as possible, I will contribute with boards. I 
could also add our custom boards, there are at least 3.


but in advance, please, just make me run a script that fetches 
build/execute tasks with some python on simple tool.


If I need to deploy some docker or some niche CI tool, this is much,much 
less likely to happen.


Sebastien

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[raiden00pl]

> Quite the opposite :-) Testing few boards multiple times in the same
by different people may be a bit of waste, but its more about anyone
providing what they have at hand and I am sure different people from
around the world will have different boards, more people more boards
coverage, and we will verify different boards configuration in real
world :-)

NOPE. Many boards are just copy-paste of other boards. Most stm32 boards
look like this. Chips with the same architecture use the same code,
so when we test a more advanced chip, we also test the code for a less
advanced chip.

The flaw in your reasoning is that we don't have many people around the
world,
just a few, and even fewer, who are willing to get involved in this project
:)
Another thing is that some of the boards are obsolete and impossible to get.

Testing everything as a final goal is OK, but right now it's a waste of
resources.
I think it's much more productive to select the minimum number of boards
that give
the greatest coverage for the code.

> It seems it is already spread over many various emails and github
issues. I think emails are good for discussions, while github is good
for noting conclusions down. But if you like Confluence we may try it
why not :-)

This is exactly the problem with email. This is not the first time these
topics are
discussed. Similar discussions and ideas have already been here. Where are
they now? Somewhere in dozens of emails over the years that I can't even
find.
Email for discussion and decision-making - OK, email as a knowledge base
and
an organized place with content - not really.

czw., 6 lut 2025 o 04:09 Tomek CEDRO  napisał(a):

> On Wed, Feb 5, 2025 at 8:25 AM raiden00pl  wrote:
> > But first we should determine what things we want to test, not what
> boards.
> > Knowing what things we want to test, we can design test cases and
> possibly
> > use what is already available.
>
> This will be part of the drunx architecture, but I have created a
> dedicated Issue just for DRUNX Test Scenarios :-)
>
> https://github.com/apache/nuttx/issues/15773
>
> Thanks! :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

On Wed, Feb 5, 2025 at 8:25 AM raiden00pl  wrote:
> But first we should determine what things we want to test, not what boards.
> Knowing what things we want to test, we can design test cases and possibly
> use what is already available.

This will be part of the drunx architecture, but I have created a
dedicated Issue just for DRUNX Test Scenarios :-)

https://github.com/apache/nuttx/issues/15773

Thanks! :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

On Wed, Feb 5, 2025 at 3:12 PM Sebastien Lorquet  wrote:
> This project is not even started yet :-)

Ekhem, its here already for some months in that form or another (see
Lup's CI and Dashboard, see my wall), not corporate style, not wearing
suits and expensive watches, not yet as we want, not yet as we need,
but man, small steps we do what we can with what we have :D :D

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

On Wed, Feb 5, 2025 at 8:25 AM raiden00pl  wrote:
> As mentioned earlier, testing all boards is pointless, especially since the
> project
> has very limited resources. Choosing a few boards that will allow us to
> test as many
> things as possible is the most optimal approach.

Quite the opposite :-) Testing few boards multiple times in the same
by different people may be a bit of waste, but its more about anyone
providing what they have at hand and I am sure different people from
around the world will have different boards, more people more boards
coverage, and we will verify different boards configuration in real
world :-)

This will provide redundancy to available tests, because some folks
may be offline, on vacation, changing location, etc, what if they had
boards that we need? Someone else will provide the results :-)

This will verify existing boards configuration and potential issues
that will lead to a fix.

This will attract new people to NuttX just because they have spare
unused hardware and some free time to play.

This will attract new people to NuttX because people will have boards
that are not yet supported so this may be a great motivator to learn
NuttX hands-on :-)

etc :-)


> But first we should determine what things we want to test, not what boards.
> Knowing what things we want to test, we can design test cases and possibly
> use what is already available.

Yes thus my previous questions in different email threads and I did
put that question already on github issue, updated to:

We need a building blocks for base / extended / specific / custom test
scenarios. Base tests will be mandatory and cover all boards (i.e. nsh
+ help + ostest). Extended may include benchmarks, stress tests, will
be optional (may not be possible on small platforms) but may provide
additional results like performance improvement or degradation.
Specific tests will be optional too and would cover arch / board
specific tests. There must be a way to implement Custom scenarios for
closed testing of custom hardware etc.


> But e-mail and github don't seem to me to be a good tool for brainstorming
> and ideas. Maybe Confluence pages would be better? I haven't used
> Confluence
> for a few years, I just hope it's not as slow as it used to be :)
>
> I can create a Confluence page and describe my ideas about testing, I have
> some
> thoughts about NuttX testing written somewhere in my private notes. Others
> can do
> the same.

I like the idea but... :D .. this will be yet another platform to
spread information.. and it never really helper in projects I was part
of except these whole projects were fully and only conducted on
Confluence :-)

It seems it is already spread over many various emails and github
issues. I think emails are good for discussions, while github is good
for noting conclusions down. But if you like Confluence we may try it
why not :-)


> Email can be good for decision making and maybe gathering more feedback from
> the community, but it's a shitty tool for more complex work. Or I'm too
> young
> to use it comfortably :P

Over the years there were hundreds of platforms, their number is
increasing, most of them are long no more, while email sill works, its
small, still using letters, read it when you want, it wont disappear,
your account will not be banned, easy to backup, and you have
everything in one place :-)

Thank you Raiden! All constructive criticism is welcome, this leads to
even better solution :-)

Tomek

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Sebastien Lorquet]

I would say testing as much as possible is always a benefit.

I have worked for more than 15 years as an embedded specialist managing 
mission critical code, and every bug we found later in the project was 
in a test gap.


As my boss usually say, "If it's not tested, it doesnt work" and this 
has been universally true.


The actual issue is that nuttx has very few resources, and that one is a 
real blocker.


So starting small and improving later is the only realistic way forward.


Please, dont add more tools to discuss more issues, the only result will 
be more dilution of the information. I have never seen any project where 
more communication channels has had benefit, quite the contrary.


Email is actually fine, its not perfect, but it has the merit to exist 
and be usable *today* without additional work.


This project is not even started yet :-)


What do you want to test? Do the minimum. ostest, hello world. Build a 
simple infra that can automatically build a board and run a hello world 
on it. Bonus if everyone can self host it by running very few code. 
avoid docker and modern heavy tools. This can be a simple python script 
using pyserial and openocd/esptool/whatever is required for a platform.


This is a very valid and useful first step, it will catch many bugs that 
cannot be seen by building only.


You will add SPI and I2C and all the desirable bell and whistles later.


But I maintain this: Having hello world run on an actual board 
automatically is 90% of the work. and it's already useful.



Sebastien


On 05/02/2025 08:22, raiden00pl wrote:

As mentioned earlier, testing all boards is pointless, especially since the
project
has very limited resources. Choosing a few boards that will allow us to
test as many
things as possible is the most optimal approach.

But first we should determine what things we want to test, not what boards.
Knowing what things we want to test, we can design test cases and possibly
use what is already available.

But e-mail and github don't seem to me to be a good tool for brainstorming
and ideas. Maybe Confluence pages would be better? I haven't used
Confluence
for a few years, I just hope it's not as slow as it used to be :)

I can create a Confluence page and describe my ideas about testing, I have
some
thoughts about NuttX testing written somewhere in my private notes. Others
can do
the same.

Email can be good for decision making and maybe gathering more feedback from
the community, but it's a shitty tool for more complex work. Or I'm too
young
to use it comfortably :P

wt., 4 lut 2025 o 12:03 Tomek CEDRO  napisał(a):


On Tue, Feb 4, 2025 at 11:42 AM Luchian Mihai 
wrote:

Hi!
First thing, I'm fairly new to nuttx so I might be off subject but here

is

my hot take on this subject.

Welcome and have fun Mihai! :-)



NuttX is offering support for a lot of boards, more than what DRUNX

should

require.

Eg. stm32f3 family, offering support for all the boards would benefit the
boards more than the NuttX codebase.
These boards share 80% of the code but each in its own files, so most
differences are due to lack of backporting.

My suggestion is to take a single board for each mcu family as mandatory
NuttX support, others as optional.
I'm not saying to drop support, or offer less support for those, just to
treat the mandatory as higher prio.
For the moment we can choose what is mandatory, and at a later time, when
DRUNX would be stable, move the optional ones to DRUNX repo (for

example).

The idea is that everyone can test what they have at hand and then
gather the results, that should sum up to full board list one day :-)
Also different people will use different build hosts, different
compilers, etc, so even if the same board is tested in different build
environment that can also reveal potential issues to be fixed :-)

Yes, for sure we need at least one board from each family for start,
then adding more boards, we all do release testing that way or
another, by hand or scripted, we now have to find a way to make it
distributed easy to setup fire-and-forget :-)



TLDR: I think less is more, less "official" support, more "official"
coverage.

More means more. Less means less. Lets keep thing simple in this
inverted world where word have no meaning anymore :D :D :D

Its a small project based on voluntary work with zero financial
support from big companies. We will define a testing architecture for
sure, but for now its a fresh concept and each one of us try different
area to create small building blocks that will give us the solution we
need, one day, hopefully ;-)

Please go ahead Mihai and try your boards, start with one that you
know best, use PyTest to create build and runtime automation, create
"selftest" board config that will run a script with `uname -a; help;
ostest, coremark`, gather the logs, parse the results, see how
nuttx-dashboard works based on gist, see what problems we have, see
how can we solve them hands-on :-) Maybe there is something better

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Simon Filgis]

or starting to implement a monster that only prevents anyone from changing.
"No change anymore" can be implemented easier, just stop working and go on
vacation, haha. But no joke, that's also a way to send a project to death.

--
Hard- and Softwaredevelopment Consultant

Geschäftsführung: Simon Filgis
USt-IdNr.: DE305343278
ISO9001:2015 


On Wed, Feb 5, 2025 at 2:39 PM Simon Filgis 
wrote:

> Dear all,
>
> each arch, driver and app tested once would be enough I think. A matrix
> can help to identify test-gaps. Double testing is nice and triple testing
> is not of benefit any more.
>
> The goal should be to have fast access to results with transparency. I
> fear starting to maintain a useless monster ;)
>
> Simon
>
> --
> Hard- and Softwaredevelopment Consultant
>
> Geschäftsführung: Simon Filgis
> USt-IdNr.: DE305343278
> ISO9001:2015 
>
>
> On Wed, Feb 5, 2025 at 1:27 PM Alan C. Assis  wrote:
>
>> I think we should test the most complete/complex boards from each arch. It
>> will cover most of the issues that could after other boards.
>>
>> BR,
>>
>> Alan
>>
>> On Wed, Feb 5, 2025 at 4:23 AM raiden00pl  wrote:
>>
>> > As mentioned earlier, testing all boards is pointless, especially since
>> the
>> > project
>> > has very limited resources. Choosing a few boards that will allow us to
>> > test as many
>> > things as possible is the most optimal approach.
>> >
>> > But first we should determine what things we want to test, not what
>> boards.
>> > Knowing what things we want to test, we can design test cases and
>> possibly
>> > use what is already available.
>> >
>> > But e-mail and github don't seem to me to be a good tool for
>> brainstorming
>> > and ideas. Maybe Confluence pages would be better? I haven't used
>> > Confluence
>> > for a few years, I just hope it's not as slow as it used to be :)
>> >
>> > I can create a Confluence page and describe my ideas about testing, I
>> have
>> > some
>> > thoughts about NuttX testing written somewhere in my private notes.
>> Others
>> > can do
>> > the same.
>> >
>> > Email can be good for decision making and maybe gathering more feedback
>> > from
>> > the community, but it's a shitty tool for more complex work. Or I'm too
>> > young
>> > to use it comfortably :P
>> >
>> > wt., 4 lut 2025 o 12:03 Tomek CEDRO  napisał(a):
>> >
>> > > On Tue, Feb 4, 2025 at 11:42 AM Luchian Mihai <
>> luchiann.mi...@gmail.com>
>> > > wrote:
>> > > > Hi!
>> > > > First thing, I'm fairly new to nuttx so I might be off subject but
>> here
>> > > is
>> > > > my hot take on this subject.
>> > >
>> > > Welcome and have fun Mihai! :-)
>> > >
>> > >
>> > > > NuttX is offering support for a lot of boards, more than what DRUNX
>> > > should
>> > > > require.
>> > > >
>> > > > Eg. stm32f3 family, offering support for all the boards would
>> benefit
>> > the
>> > > > boards more than the NuttX codebase.
>> > > > These boards share 80% of the code but each in its own files, so
>> most
>> > > > differences are due to lack of backporting.
>> > > >
>> > > > My suggestion is to take a single board for each mcu family as
>> > mandatory
>> > > > NuttX support, others as optional.
>> > > > I'm not saying to drop support, or offer less support for those,
>> just
>> > to
>> > > > treat the mandatory as higher prio.
>> > > > For the moment we can choose what is mandatory, and at a later time,
>> > when
>> > > > DRUNX would be stable, move the optional ones to DRUNX repo (for
>> > > example).
>> > >
>> > > The idea is that everyone can test what they have at hand and then
>> > > gather the results, that should sum up to full board list one day :-)
>> > > Also different people will use different build hosts, different
>> > > compilers, etc, so even if the same board is tested in different build
>> > > environment that can also reveal potential issues to be fixed :-)
>> > >
>> > > Yes, for sure we need at least one board from each family for start,
>> > > then adding more boards, we all do release testing that way or
>> > > another, by hand or scripted, we now have to find a way to make it
>> > > distributed easy to setup fire-and-forget :-)
>> > >
>> > >
>> > > > TLDR: I think less is more, less "official" support, more "official"
>> > > > coverage.
>> > >
>> > > More means more. Less means less. Lets keep thing simple in this
>> > > inverted world where word have no meaning anymore :D :D :D
>> > >
>> > > Its a small project based on voluntary work with zero financial
>> > > support from big companies. We will define a testing architecture for
>> > > sure, but for now its a fresh concept and each one of us try different
>> > > area to create small building blocks that will give us the solution we
>> > > need, one day, hopefully ;-)
>> > >
>> > > Please go ahead Mihai and try your boards, start with one that you
>> > > know best, use PyTest to create build and runtime automation, creat

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Simon Filgis]

Dear all,

each arch, driver and app tested once would be enough I think. A matrix can
help to identify test-gaps. Double testing is nice and triple testing is
not of benefit any more.

The goal should be to have fast access to results with transparency. I fear
starting to maintain a useless monster ;)

Simon

--
Hard- and Softwaredevelopment Consultant

Geschäftsführung: Simon Filgis
USt-IdNr.: DE305343278
ISO9001:2015 


On Wed, Feb 5, 2025 at 1:27 PM Alan C. Assis  wrote:

> I think we should test the most complete/complex boards from each arch. It
> will cover most of the issues that could after other boards.
>
> BR,
>
> Alan
>
> On Wed, Feb 5, 2025 at 4:23 AM raiden00pl  wrote:
>
> > As mentioned earlier, testing all boards is pointless, especially since
> the
> > project
> > has very limited resources. Choosing a few boards that will allow us to
> > test as many
> > things as possible is the most optimal approach.
> >
> > But first we should determine what things we want to test, not what
> boards.
> > Knowing what things we want to test, we can design test cases and
> possibly
> > use what is already available.
> >
> > But e-mail and github don't seem to me to be a good tool for
> brainstorming
> > and ideas. Maybe Confluence pages would be better? I haven't used
> > Confluence
> > for a few years, I just hope it's not as slow as it used to be :)
> >
> > I can create a Confluence page and describe my ideas about testing, I
> have
> > some
> > thoughts about NuttX testing written somewhere in my private notes.
> Others
> > can do
> > the same.
> >
> > Email can be good for decision making and maybe gathering more feedback
> > from
> > the community, but it's a shitty tool for more complex work. Or I'm too
> > young
> > to use it comfortably :P
> >
> > wt., 4 lut 2025 o 12:03 Tomek CEDRO  napisał(a):
> >
> > > On Tue, Feb 4, 2025 at 11:42 AM Luchian Mihai <
> luchiann.mi...@gmail.com>
> > > wrote:
> > > > Hi!
> > > > First thing, I'm fairly new to nuttx so I might be off subject but
> here
> > > is
> > > > my hot take on this subject.
> > >
> > > Welcome and have fun Mihai! :-)
> > >
> > >
> > > > NuttX is offering support for a lot of boards, more than what DRUNX
> > > should
> > > > require.
> > > >
> > > > Eg. stm32f3 family, offering support for all the boards would benefit
> > the
> > > > boards more than the NuttX codebase.
> > > > These boards share 80% of the code but each in its own files, so most
> > > > differences are due to lack of backporting.
> > > >
> > > > My suggestion is to take a single board for each mcu family as
> > mandatory
> > > > NuttX support, others as optional.
> > > > I'm not saying to drop support, or offer less support for those, just
> > to
> > > > treat the mandatory as higher prio.
> > > > For the moment we can choose what is mandatory, and at a later time,
> > when
> > > > DRUNX would be stable, move the optional ones to DRUNX repo (for
> > > example).
> > >
> > > The idea is that everyone can test what they have at hand and then
> > > gather the results, that should sum up to full board list one day :-)
> > > Also different people will use different build hosts, different
> > > compilers, etc, so even if the same board is tested in different build
> > > environment that can also reveal potential issues to be fixed :-)
> > >
> > > Yes, for sure we need at least one board from each family for start,
> > > then adding more boards, we all do release testing that way or
> > > another, by hand or scripted, we now have to find a way to make it
> > > distributed easy to setup fire-and-forget :-)
> > >
> > >
> > > > TLDR: I think less is more, less "official" support, more "official"
> > > > coverage.
> > >
> > > More means more. Less means less. Lets keep thing simple in this
> > > inverted world where word have no meaning anymore :D :D :D
> > >
> > > Its a small project based on voluntary work with zero financial
> > > support from big companies. We will define a testing architecture for
> > > sure, but for now its a fresh concept and each one of us try different
> > > area to create small building blocks that will give us the solution we
> > > need, one day, hopefully ;-)
> > >
> > > Please go ahead Mihai and try your boards, start with one that you
> > > know best, use PyTest to create build and runtime automation, create
> > > "selftest" board config that will run a script with `uname -a; help;
> > > ostest, coremark`, gather the logs, parse the results, see how
> > > nuttx-dashboard works based on gist, see what problems we have, see
> > > how can we solve them hands-on :-) Maybe there is something better
> > > than PyTest. Maybe there are other ways. You try it out and share the
> > > feedback :-)
> > >
> > > If this is too much, use smaller steps, think of it as automation tool
> > > for release testing on different boards :-)
> > >
> > > Thank you and take care :-)
> > > Tomek
> > >
> > > --
> > > CeDeROM

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Alan C. Assis]

I think we should test the most complete/complex boards from each arch. It
will cover most of the issues that could after other boards.

BR,

Alan

On Wed, Feb 5, 2025 at 4:23 AM raiden00pl  wrote:

> As mentioned earlier, testing all boards is pointless, especially since the
> project
> has very limited resources. Choosing a few boards that will allow us to
> test as many
> things as possible is the most optimal approach.
>
> But first we should determine what things we want to test, not what boards.
> Knowing what things we want to test, we can design test cases and possibly
> use what is already available.
>
> But e-mail and github don't seem to me to be a good tool for brainstorming
> and ideas. Maybe Confluence pages would be better? I haven't used
> Confluence
> for a few years, I just hope it's not as slow as it used to be :)
>
> I can create a Confluence page and describe my ideas about testing, I have
> some
> thoughts about NuttX testing written somewhere in my private notes. Others
> can do
> the same.
>
> Email can be good for decision making and maybe gathering more feedback
> from
> the community, but it's a shitty tool for more complex work. Or I'm too
> young
> to use it comfortably :P
>
> wt., 4 lut 2025 o 12:03 Tomek CEDRO  napisał(a):
>
> > On Tue, Feb 4, 2025 at 11:42 AM Luchian Mihai 
> > wrote:
> > > Hi!
> > > First thing, I'm fairly new to nuttx so I might be off subject but here
> > is
> > > my hot take on this subject.
> >
> > Welcome and have fun Mihai! :-)
> >
> >
> > > NuttX is offering support for a lot of boards, more than what DRUNX
> > should
> > > require.
> > >
> > > Eg. stm32f3 family, offering support for all the boards would benefit
> the
> > > boards more than the NuttX codebase.
> > > These boards share 80% of the code but each in its own files, so most
> > > differences are due to lack of backporting.
> > >
> > > My suggestion is to take a single board for each mcu family as
> mandatory
> > > NuttX support, others as optional.
> > > I'm not saying to drop support, or offer less support for those, just
> to
> > > treat the mandatory as higher prio.
> > > For the moment we can choose what is mandatory, and at a later time,
> when
> > > DRUNX would be stable, move the optional ones to DRUNX repo (for
> > example).
> >
> > The idea is that everyone can test what they have at hand and then
> > gather the results, that should sum up to full board list one day :-)
> > Also different people will use different build hosts, different
> > compilers, etc, so even if the same board is tested in different build
> > environment that can also reveal potential issues to be fixed :-)
> >
> > Yes, for sure we need at least one board from each family for start,
> > then adding more boards, we all do release testing that way or
> > another, by hand or scripted, we now have to find a way to make it
> > distributed easy to setup fire-and-forget :-)
> >
> >
> > > TLDR: I think less is more, less "official" support, more "official"
> > > coverage.
> >
> > More means more. Less means less. Lets keep thing simple in this
> > inverted world where word have no meaning anymore :D :D :D
> >
> > Its a small project based on voluntary work with zero financial
> > support from big companies. We will define a testing architecture for
> > sure, but for now its a fresh concept and each one of us try different
> > area to create small building blocks that will give us the solution we
> > need, one day, hopefully ;-)
> >
> > Please go ahead Mihai and try your boards, start with one that you
> > know best, use PyTest to create build and runtime automation, create
> > "selftest" board config that will run a script with `uname -a; help;
> > ostest, coremark`, gather the logs, parse the results, see how
> > nuttx-dashboard works based on gist, see what problems we have, see
> > how can we solve them hands-on :-) Maybe there is something better
> > than PyTest. Maybe there are other ways. You try it out and share the
> > feedback :-)
> >
> > If this is too much, use smaller steps, think of it as automation tool
> > for release testing on different boards :-)
> >
> > Thank you and take care :-)
> > Tomek
> >
> > --
> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[raiden00pl]

As mentioned earlier, testing all boards is pointless, especially since the
project
has very limited resources. Choosing a few boards that will allow us to
test as many
things as possible is the most optimal approach.

But first we should determine what things we want to test, not what boards.
Knowing what things we want to test, we can design test cases and possibly
use what is already available.

But e-mail and github don't seem to me to be a good tool for brainstorming
and ideas. Maybe Confluence pages would be better? I haven't used
Confluence
for a few years, I just hope it's not as slow as it used to be :)

I can create a Confluence page and describe my ideas about testing, I have
some
thoughts about NuttX testing written somewhere in my private notes. Others
can do
the same.

Email can be good for decision making and maybe gathering more feedback from
the community, but it's a shitty tool for more complex work. Or I'm too
young
to use it comfortably :P

wt., 4 lut 2025 o 12:03 Tomek CEDRO  napisał(a):

> On Tue, Feb 4, 2025 at 11:42 AM Luchian Mihai 
> wrote:
> > Hi!
> > First thing, I'm fairly new to nuttx so I might be off subject but here
> is
> > my hot take on this subject.
>
> Welcome and have fun Mihai! :-)
>
>
> > NuttX is offering support for a lot of boards, more than what DRUNX
> should
> > require.
> >
> > Eg. stm32f3 family, offering support for all the boards would benefit the
> > boards more than the NuttX codebase.
> > These boards share 80% of the code but each in its own files, so most
> > differences are due to lack of backporting.
> >
> > My suggestion is to take a single board for each mcu family as mandatory
> > NuttX support, others as optional.
> > I'm not saying to drop support, or offer less support for those, just to
> > treat the mandatory as higher prio.
> > For the moment we can choose what is mandatory, and at a later time, when
> > DRUNX would be stable, move the optional ones to DRUNX repo (for
> example).
>
> The idea is that everyone can test what they have at hand and then
> gather the results, that should sum up to full board list one day :-)
> Also different people will use different build hosts, different
> compilers, etc, so even if the same board is tested in different build
> environment that can also reveal potential issues to be fixed :-)
>
> Yes, for sure we need at least one board from each family for start,
> then adding more boards, we all do release testing that way or
> another, by hand or scripted, we now have to find a way to make it
> distributed easy to setup fire-and-forget :-)
>
>
> > TLDR: I think less is more, less "official" support, more "official"
> > coverage.
>
> More means more. Less means less. Lets keep thing simple in this
> inverted world where word have no meaning anymore :D :D :D
>
> Its a small project based on voluntary work with zero financial
> support from big companies. We will define a testing architecture for
> sure, but for now its a fresh concept and each one of us try different
> area to create small building blocks that will give us the solution we
> need, one day, hopefully ;-)
>
> Please go ahead Mihai and try your boards, start with one that you
> know best, use PyTest to create build and runtime automation, create
> "selftest" board config that will run a script with `uname -a; help;
> ostest, coremark`, gather the logs, parse the results, see how
> nuttx-dashboard works based on gist, see what problems we have, see
> how can we solve them hands-on :-) Maybe there is something better
> than PyTest. Maybe there are other ways. You try it out and share the
> feedback :-)
>
> If this is too much, use smaller steps, think of it as automation tool
> for release testing on different boards :-)
>
> Thank you and take care :-)
> Tomek
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Tomek CEDRO]

On Tue, Feb 4, 2025 at 11:42 AM Luchian Mihai  wrote:
> Hi!
> First thing, I'm fairly new to nuttx so I might be off subject but here is
> my hot take on this subject.

Welcome and have fun Mihai! :-)


> NuttX is offering support for a lot of boards, more than what DRUNX should
> require.
>
> Eg. stm32f3 family, offering support for all the boards would benefit the
> boards more than the NuttX codebase.
> These boards share 80% of the code but each in its own files, so most
> differences are due to lack of backporting.
>
> My suggestion is to take a single board for each mcu family as mandatory
> NuttX support, others as optional.
> I'm not saying to drop support, or offer less support for those, just to
> treat the mandatory as higher prio.
> For the moment we can choose what is mandatory, and at a later time, when
> DRUNX would be stable, move the optional ones to DRUNX repo (for example).

The idea is that everyone can test what they have at hand and then
gather the results, that should sum up to full board list one day :-)
Also different people will use different build hosts, different
compilers, etc, so even if the same board is tested in different build
environment that can also reveal potential issues to be fixed :-)

Yes, for sure we need at least one board from each family for start,
then adding more boards, we all do release testing that way or
another, by hand or scripted, we now have to find a way to make it
distributed easy to setup fire-and-forget :-)


> TLDR: I think less is more, less "official" support, more "official"
> coverage.

More means more. Less means less. Lets keep thing simple in this
inverted world where word have no meaning anymore :D :D :D

Its a small project based on voluntary work with zero financial
support from big companies. We will define a testing architecture for
sure, but for now its a fresh concept and each one of us try different
area to create small building blocks that will give us the solution we
need, one day, hopefully ;-)

Please go ahead Mihai and try your boards, start with one that you
know best, use PyTest to create build and runtime automation, create
"selftest" board config that will run a script with `uname -a; help;
ostest, coremark`, gather the logs, parse the results, see how
nuttx-dashboard works based on gist, see what problems we have, see
how can we solve them hands-on :-) Maybe there is something better
than PyTest. Maybe there are other ways. You try it out and share the
feedback :-)

If this is too much, use smaller steps, think of it as automation tool
for release testing on different boards :-)

Thank you and take care :-)
Tomek

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: [DRUNX] Distributed Runtime and bUild for NuttX

[Luchian Mihai]

Hi!
First thing, I'm fairly new to nuttx so I might be off subject but here is
my hot take on this subject.

NuttX is offering support for a lot of boards, more than what DRUNX should
require.

Eg. stm32f3 family, offering support for all the boards would benefit the
boards more than the NuttX codebase.
These boards share 80% of the code but each in its own files, so most
differences are due to lack of backporting.

My suggestion is to take a single board for each mcu family as mandatory
NuttX support, others as optional.
I'm not saying to drop support, or offer less support for those, just to
treat the mandatory as higher prio.
For the moment we can choose what is mandatory, and at a later time, when
DRUNX would be stable, move the optional ones to DRUNX repo (for example).

TLDR: I think less is more, less "official" support, more "official"
coverage.

Have a great day!
Mihai

On Tue, 4 Feb 2025 at 10:25, Tomek CEDRO  wrote:

> Hello world :-)
>
> Lets keep the distributed build and runtime test environment
> discussion in this mailing list thread.
>
> Here is the discussion issue on GitHub:
>
> https://github.com/apache/nuttx/issues/15730
>
>
> Some things for start, at this point in time:
>
> 1. If anyone has better working name please share :-)
>
> 2. Lup created nice dashboard for gathering and presentation in one place:
>
> https://nuttx-dashboard.org/
>
> This platform gather logs from builds that use the same scripts as CI
> on GitHub runs. We had the CI overuse problems on GitHub and risk of
> loosing CI, thus this solution as backup.
>
> We can extend it with runtime tests. That part is still TODO. PyTest
> seems the best and commonly used framework. This can launch sets of
> tests like build, run, gather logs, upload somewhere for analysis
> (gist -> dashboard).
>
> 3. We need to select a set of standard tests to be processed runtime.
> ostest and some benchmarks / stress tests. In a pefcest situation. We
> are working on apps/testing rework to split into apps/system
> applications, apps/testing utilities, and apps/tests scenarios that
> could run on MCU and do the tests with PASS/FAIL result list.
>
> 4. This solution should be based on zero-trust approach. That is
> upstream does not trust user testing, but we provide tools to run
> tests and report back results that would prove changes are not
> breaking stuff.
>
> 5. Not to create more mess in the nuttx repo I have created a
> dedicated workbench repo nuttx-drunx that we may work on and if
> something is ready we will send to nuttx upstream:
>
> https://github.com/cederom/nuttx-drunx
>
> Thank you :-)
> Tomek
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>