Re: CI tests (was: Re: [OT] Learning Makefiles)

[Brennan Ashton]

On Mon, May 29, 2023 at 2:41 AM Michael Gielda  wrote:
>
> Haha, no worries, just subscribed on Friday for this particular reason :) 
> glad Renode was useful for EOS S3 and of course happy to see how we can 
> enable more of that.
>
>
> You mention you have a CI generating all the elfs, could you point me to 
> where that can be found? We could grab those and try to run them, see what 
> happens.
>
>

There is a giant bundle of the build artifacts that can be downloaded
here under linux-builds
https://github.com/apache/nuttx/actions/runs/5128284656
https://github.com/apache/nuttx/suites/13258933442/artifacts/723073966

For example there should be a stm32f4discovery/nsh with a nuttx.elf
which would be a small shell build for that board.

>From what I remember a few years ago was that I had to sub out some
clock and power related code to allow the target to boot, but I have
not tried in at least a year.

--Brennan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Tomek CEDRO]

On Wed, May 31, 2023 at 5:26 AM Brennan Ashton wrote:
> Still dropped him. Apparently I cannot use email correctly today...

Beware of big changes in quota handling at Google in recent days..
especially these using Workspace (Suite or whatever name it has now)..
I just got my email back to work after seeing that it went silent..
previous 30GB limit has been decreased to 15GB and all services
silently stopped (not even quota exceeded bounce message) until I
cleared the now common account storage.. so I did a backup and removed
everything.. now it will be easier to move away from Google o_O

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Michael Gielda]

Haha, no worries, just subscribed on Friday for this particular reason :)
glad Renode was useful for EOS S3 and of course happy to see how we can
enable more of that.


You mention you have a CI generating all the elfs, could you point me to
where that can be found? We could grab those and try to run them, see what
happens.


I can explain how we approach the platform generation we are doing for
Zephyr/U-Boot now to see how we could make it work for NuttX. Basically we
have this https://github.com/antmicro/dts2repl utility which uses the
Zephyr device trees generated with its build systems - with some
hand-written overlays (
https://github.com/antmicro/dts2repl/tree/main/dts2repl/overlay) which we
try to minimize / incorporate into the Zephyr device trees - into Renode's
repl files (platform files, the ones describing HW platforms). For the
simulation scenario, or .resc files, we have a template that's standard
across all platforms for a specific Zephyr demo. Same could be done for
NuttX of course.


Then we use this to run all the tests across all the platforms, and gather
and visualize the results.


If, when running NuttX (or any other payload) on those platforms we find
some of our platforms descriptions lacking some specific stuff like power
or clock management, we would add the necessary stuff into the overlays (or
Zephyr’s dts if possible)


I think some STM32 platforms would be a good entry point, since there are
many and they are pretty well supported in Renode/Zephyr/NuttX. But again,
if you point me to the elf files you are hosting, we could probably consume
them and report the results.


CCing my team to follow the conversation. They will have more details if
needed.


Best regards,

Michael


On Sat, 27 May 2023 at 01:01, Brennan Ashton 
wrote:

> Still dropped him. Apparently I cannot use email correctly today...
>
> On Fri, May 26, 2023, 3:40 PM Brennan Ashton 
> wrote:
>
>> Adding Michael back since I responded to him and he might not be on the
>> list...
>>
>> On Fri, May 26, 2023 at 3:23 PM Brennan Ashton
>>  wrote:
>> >
>> >
>> >
>> > On Fri, May 26, 2023, 3:11 PM Michael Gielda 
>> wrote:
>> >>
>> >> Hi Alan, nice to meet you Nathan,
>> >>
>> >> Indeed, we discussed with Alan some time ago, having a Nuttx-focused
>> >> dashboard can be done and we assume would be useful. We just didn’t
>> get to
>> >> that as it would have been yet another research project (we are now
>> also
>> >> building a U-Boot dashboard, as we added 64-bit Cortex-A support -
>> >> https://antmicro.com/blog/2023/04/armv8-a-support-in-renode/ - as
>> well as
>> >> Cortex-R). But if there is interest, perhaps it’s worth revisiting the
>> idea.
>> >>
>> >> On our end, we base the simulation platform configuration on Zephyr,
>> so we
>> >> basically need a mapping of platforms in Nuttx to their “Zephyr names”
>> - we
>> >> can start of course with 1-2 and expand to more with time.
>> >>
>> >> Best regards,
>> >>
>> >> Michael
>> >
>> >
>> > Hey Michael,
>> > I actually had used renode when added upstream support a few years ago
>> with the QuickLogic EOS S3, and it was quite helpful.  That's when I talked
>> a bit about seeing if we could use it as a test platform here.
>> https://docs.google.com/presentation/d/1G1IKe3SjmB42eb7dhdoa4P_nREknsJy1/edit#slide=id.g9044e8e6ab_0_17
>> >
>> > One of the reasons I added support to export build artifacts in CI was
>> to enable having downstream stages that can pick up the already generated
>> elf.
>> >
>> > One thing at the time I hit with the targets I tested was some
>> assumptions around power and clock management that prevented NuttX from
>> booting (waiting for status bits usually).  So if we do go down this route
>> I suspect we will need to upstream some of those bits to the emulation
>> which I don't recall being too complicated.
>> >
>> > If people have a supported platform they would like to see I would be
>> happy to wire a proof of concept through.
>> >
>> > --Brennan
>>
>

-- 
Michael Gielda
mobile: +46 73 759 47 27
Antmicro AB | www.antmicro.com
Kistagången 16, 164 40 Kista, Sweden

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Brennan Ashton]

Still dropped him. Apparently I cannot use email correctly today...

On Fri, May 26, 2023, 3:40 PM Brennan Ashton 
wrote:

> Adding Michael back since I responded to him and he might not be on the
> list...
>
> On Fri, May 26, 2023 at 3:23 PM Brennan Ashton
>  wrote:
> >
> >
> >
> > On Fri, May 26, 2023, 3:11 PM Michael Gielda 
> wrote:
> >>
> >> Hi Alan, nice to meet you Nathan,
> >>
> >> Indeed, we discussed with Alan some time ago, having a Nuttx-focused
> >> dashboard can be done and we assume would be useful. We just didn’t get
> to
> >> that as it would have been yet another research project (we are now also
> >> building a U-Boot dashboard, as we added 64-bit Cortex-A support -
> >> https://antmicro.com/blog/2023/04/armv8-a-support-in-renode/ - as well
> as
> >> Cortex-R). But if there is interest, perhaps it’s worth revisiting the
> idea.
> >>
> >> On our end, we base the simulation platform configuration on Zephyr, so
> we
> >> basically need a mapping of platforms in Nuttx to their “Zephyr names”
> - we
> >> can start of course with 1-2 and expand to more with time.
> >>
> >> Best regards,
> >>
> >> Michael
> >
> >
> > Hey Michael,
> > I actually had used renode when added upstream support a few years ago
> with the QuickLogic EOS S3, and it was quite helpful.  That's when I talked
> a bit about seeing if we could use it as a test platform here.
> https://docs.google.com/presentation/d/1G1IKe3SjmB42eb7dhdoa4P_nREknsJy1/edit#slide=id.g9044e8e6ab_0_17
> >
> > One of the reasons I added support to export build artifacts in CI was
> to enable having downstream stages that can pick up the already generated
> elf.
> >
> > One thing at the time I hit with the targets I tested was some
> assumptions around power and clock management that prevented NuttX from
> booting (waiting for status bits usually).  So if we do go down this route
> I suspect we will need to upstream some of those bits to the emulation
> which I don't recall being too complicated.
> >
> > If people have a supported platform they would like to see I would be
> happy to wire a proof of concept through.
> >
> > --Brennan
>

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Brennan Ashton]

Adding Michael back since I responded to him and he might not be on the list...

On Fri, May 26, 2023 at 3:23 PM Brennan Ashton
 wrote:
>
>
>
> On Fri, May 26, 2023, 3:11 PM Michael Gielda  wrote:
>>
>> Hi Alan, nice to meet you Nathan,
>>
>> Indeed, we discussed with Alan some time ago, having a Nuttx-focused
>> dashboard can be done and we assume would be useful. We just didn’t get to
>> that as it would have been yet another research project (we are now also
>> building a U-Boot dashboard, as we added 64-bit Cortex-A support -
>> https://antmicro.com/blog/2023/04/armv8-a-support-in-renode/ - as well as
>> Cortex-R). But if there is interest, perhaps it’s worth revisiting the idea.
>>
>> On our end, we base the simulation platform configuration on Zephyr, so we
>> basically need a mapping of platforms in Nuttx to their “Zephyr names” - we
>> can start of course with 1-2 and expand to more with time.
>>
>> Best regards,
>>
>> Michael
>
>
> Hey Michael,
> I actually had used renode when added upstream support a few years ago with 
> the QuickLogic EOS S3, and it was quite helpful.  That's when I talked a bit 
> about seeing if we could use it as a test platform here. 
> https://docs.google.com/presentation/d/1G1IKe3SjmB42eb7dhdoa4P_nREknsJy1/edit#slide=id.g9044e8e6ab_0_17
>
> One of the reasons I added support to export build artifacts in CI was to 
> enable having downstream stages that can pick up the already generated elf.
>
> One thing at the time I hit with the targets I tested was some assumptions 
> around power and clock management that prevented NuttX from booting (waiting 
> for status bits usually).  So if we do go down this route I suspect we will 
> need to upstream some of those bits to the emulation which I don't recall 
> being too complicated.
>
> If people have a supported platform they would like to see I would be happy 
> to wire a proof of concept through.
>
> --Brennan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Brennan Ashton]

On Fri, May 26, 2023 at 3:11 PM Michael Gielda  wrote:
>
> Hi Alan, nice to meet you Nathan,
>
> Indeed, we discussed with Alan some time ago, having a Nuttx-focused
> dashboard can be done and we assume would be useful. We just didn’t get to
> that as it would have been yet another research project (we are now also
> building a U-Boot dashboard, as we added 64-bit Cortex-A support -
> https://antmicro.com/blog/2023/04/armv8-a-support-in-renode/ - as well as
> Cortex-R). But if there is interest, perhaps it’s worth revisiting the idea.
>
> On our end, we base the simulation platform configuration on Zephyr, so we
> basically need a mapping of platforms in Nuttx to their “Zephyr names” - we
> can start of course with 1-2 and expand to more with time.
>
> Best regards,
>
> Michael
>
Hey Michael,
I actually had used renode when adding upstream support a few years
ago with the QuickLogic EOS S3, and it was quite helpful.  That's when
I talked a bit about seeing if we could use it as a test platform
here.

One of the reasons I added support to export build artifacts in CI was
to enable having downstream stages that can pick up the already
generated elf.

One thing at the time I hit with the targets I tested was some
assumptions around power and clock management that prevented NuttX
from booting (waiting for status bits usually).  So if we do go down
this route I suspect we will need to upstream some of those bits to
the emulation which I don't recall being too complicated.

If people have a supported platform they would like to see I would be
happy to wire a proof of concept through.

--Brennan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Brennan Ashton]

On Fri, May 26, 2023, 3:11 PM Michael Gielda  wrote:

> Hi Alan, nice to meet you Nathan,
>
> Indeed, we discussed with Alan some time ago, having a Nuttx-focused
> dashboard can be done and we assume would be useful. We just didn’t get to
> that as it would have been yet another research project (we are now also
> building a U-Boot dashboard, as we added 64-bit Cortex-A support -
> https://antmicro.com/blog/2023/04/armv8-a-support-in-renode/ - as well as
> Cortex-R). But if there is interest, perhaps it’s worth revisiting the
> idea.
>
> On our end, we base the simulation platform configuration on Zephyr, so we
> basically need a mapping of platforms in Nuttx to their “Zephyr names” - we
> can start of course with 1-2 and expand to more with time.
>
> Best regards,
>
> Michael
>

Hey Michael,
I actually had used renode when added upstream support a few years ago with
the QuickLogic EOS S3, and it was quite helpful.  That's when I talked a
bit about seeing if we could use it as a test platform here.

One of the reason I added support to export build artifacts in CI was to
enable having downstream stages that can pick up the already generated elf.

One thing at the time I hit with the targets I tested was some assumptions
around power and clock management that prevented NuttX from booting
(waiting for status bits usually).  So if we do go down this route I
suspect we will need to upstream some of those bits to the emulation which
I don't recall being too complicated.

If people have a supported platform they would like to see I would be happy
to wire a proof of concept through.

--Brennan

>

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Michael Gielda]

Hi Alan, nice to meet you Nathan,

Indeed, we discussed with Alan some time ago, having a Nuttx-focused
dashboard can be done and we assume would be useful. We just didn’t get to
that as it would have been yet another research project (we are now also
building a U-Boot dashboard, as we added 64-bit Cortex-A support -
https://antmicro.com/blog/2023/04/armv8-a-support-in-renode/ - as well as
Cortex-R). But if there is interest, perhaps it’s worth revisiting the idea.

On our end, we base the simulation platform configuration on Zephyr, so we
basically need a mapping of platforms in Nuttx to their “Zephyr names” - we
can start of course with 1-2 and expand to more with time.

Best regards,

Michael


On Tue, 23 May 2023 at 16:42, Alan C. Assis  wrote:

> On 5/23/23, Brennan Ashton  wrote:
> > On Tue, May 23, 2023, 5:05 AM Nathan Hartman 
> > wrote:
> >
> >> On Tue, May 23, 2023 at 6:12 AM Brennan Ashton
> >>  wrote:
> >> > I have also asked in the past about cutting down on the amount of
> >> > configs
> >> > we have checked in to be something like
> >> >
> >> > board:nsh -- only nsh and somewhat small
> >> > board:jumbo -- nsh, plus as many features as can fit and are
> >> > interesting
> >> in
> >> > the platform.
> >> >
> >> > For sim and some other targets it would make sense to have more
> >> > targets,
> >> > but not for every board.
> >>
> >>
> >> The idea of "board:jumbo" is very similar to what I was saying
> >> earlier. Maybe it will allow us to test fewer boards in less time but
> >> still get better test coverage. I am in favor of *better* test
> >> coverage, not less test coverage!!
> >>
> >> In the past, we talked about having some tests in CI for each PR, and
> >> then a bigger nightly test that builds all boards/configs like Greg
> >> used to do before releases. I don't think that ever happened, but ASF
> >> has a build farm separate from GitHub that we might use, or we could
> >> request from INFRA a virtual machine to set up a complete environment.
> >> Maybe that's something to think about.
> >>
> >
> >
> > I'm not sure why we would need anything new? We can still run this in
> > GitHub actions, but generally I don't think we should be having PRs merge
> > that are not passing build tests.
> >
> >
> > As for more testing of system on boards, QEMU is great for some tests and
> > there is a thin framework that does some of that work that Xiang and
> others
> > have started.  A few years ago I also gave a talk to see if there was
> > interest in working with the folks a renode.io. Their open source
> simulator
> > is what Zypher is using and at the time had minimal support, but check
> out
> > this awesome dashboard.
> >
> > https://zephyr-dashboard.renode.io/
> >
> >
> > It would be really cool if we could join forces a bit and continue to
> build
> > off that effort and improve some of the emulation as needed (some work is
> > required).
> >
>
> Some time ago I contacted Michael (from Antmicro, Renode. I CC he
> here) and he was very prone to help, but we didn't get a dashboard for
> NuttX yet.
>
> I totally agree with that idea, maybe we could create a page that will
> show up at nuttx-dashboard.renode.io this way they don't need to
> maintain it, since they already have much work with Zephyr.
>
> > Nothing will beat hardware, but as Xiang said let's start with the easy
> bit
> > which is software.
> >
>
> For sure! Let to start with QEMU/Renode and then will add real HW support.
>
> BR,
>
> Alan
>


-- 
Michael Gielda
mobile: +46 73 759 47 27
Antmicro AB | www.antmicro.com
Kistagången 16, 164 40 Kista, Sweden

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Xiang Xiao]

Here is the script to run the basic test:
https://github.com/apache/nuttx/blob/master/tools/ci/cirun.sh
Add a link in your config:
https://github.com/apache/nuttx/blob/master/boards/risc-v/qemu-rv/rv-virt/configs/citest/run
Build script will run it and report the result after the build pass:
https://github.com/apache/nuttx/blob/master/tools/testbuild.sh#L369C1-L380

On Wed, May 24, 2023 at 3:35 AM Nathan Hartman 
wrote:

> On Tue, May 23, 2023 at 10:06 AM Brennan Ashton
>  wrote:
> >
> > On Tue, May 23, 2023, 5:05 AM Nathan Hartman 
> > wrote:
> >
> > > On Tue, May 23, 2023 at 6:12 AM Brennan Ashton
> > >  wrote:
> > > > I have also asked in the past about cutting down on the amount of
> configs
> > > > we have checked in to be something like
> > > >
> > > > board:nsh -- only nsh and somewhat small
> > > > board:jumbo -- nsh, plus as many features as can fit and are
> interesting
> > > in
> > > > the platform.
> > > >
> > > > For sim and some other targets it would make sense to have more
> targets,
> > > > but not for every board.
> > >
> > >
> > > The idea of "board:jumbo" is very similar to what I was saying
> > > earlier. Maybe it will allow us to test fewer boards in less time but
> > > still get better test coverage. I am in favor of *better* test
> > > coverage, not less test coverage!!
> > >
> > > In the past, we talked about having some tests in CI for each PR, and
> > > then a bigger nightly test that builds all boards/configs like Greg
> > > used to do before releases. I don't think that ever happened, but ASF
> > > has a build farm separate from GitHub that we might use, or we could
> > > request from INFRA a virtual machine to set up a complete environment.
> > > Maybe that's something to think about.
> > >
> >
> >
> > I'm not sure why we would need anything new? We can still run this in
> > GitHub actions, but generally I don't think we should be having PRs merge
> > that are not passing build tests.
> >
> >
> > As for more testing of system on boards, QEMU is great for some tests and
> > there is a thin framework that does some of that work that Xiang and
> others
> > have started.  A few years ago I also gave a talk to see if there was
> > interest in working with the folks a renode.io. Their open source
> simulator
> > is what Zypher is using and at the time had minimal support, but check
> out
> > this awesome dashboard.
> >
> > https://zephyr-dashboard.renode.io/
> >
> >
> > It would be really cool if we could join forces a bit and continue to
> build
> > off that effort and improve some of the emulation as needed (some work is
> > required).
>
>
> Wow, that is cool! I must have missed it when you mentioned it
> previously. Yes, this is something the NuttX project should look into.
>
> I agree with the QEMU idea as well, for those who want to build a
> test/development setup they can run locally.
>
> Both of these ideas are very good and we should pursue them. If we do
> QEMU, we should document it, or script it, or both, so that other
> community members can reproduce it and run a test system locally.
> Personally I would like such a setup, and I am interested in helping
> to document it.
>
> Cheers,
> Nathan
>

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Nathan Hartman]

On Tue, May 23, 2023 at 10:06 AM Brennan Ashton
 wrote:
>
> On Tue, May 23, 2023, 5:05 AM Nathan Hartman 
> wrote:
>
> > On Tue, May 23, 2023 at 6:12 AM Brennan Ashton
> >  wrote:
> > > I have also asked in the past about cutting down on the amount of configs
> > > we have checked in to be something like
> > >
> > > board:nsh -- only nsh and somewhat small
> > > board:jumbo -- nsh, plus as many features as can fit and are interesting
> > in
> > > the platform.
> > >
> > > For sim and some other targets it would make sense to have more targets,
> > > but not for every board.
> >
> >
> > The idea of "board:jumbo" is very similar to what I was saying
> > earlier. Maybe it will allow us to test fewer boards in less time but
> > still get better test coverage. I am in favor of *better* test
> > coverage, not less test coverage!!
> >
> > In the past, we talked about having some tests in CI for each PR, and
> > then a bigger nightly test that builds all boards/configs like Greg
> > used to do before releases. I don't think that ever happened, but ASF
> > has a build farm separate from GitHub that we might use, or we could
> > request from INFRA a virtual machine to set up a complete environment.
> > Maybe that's something to think about.
> >
>
>
> I'm not sure why we would need anything new? We can still run this in
> GitHub actions, but generally I don't think we should be having PRs merge
> that are not passing build tests.
>
>
> As for more testing of system on boards, QEMU is great for some tests and
> there is a thin framework that does some of that work that Xiang and others
> have started.  A few years ago I also gave a talk to see if there was
> interest in working with the folks a renode.io. Their open source simulator
> is what Zypher is using and at the time had minimal support, but check out
> this awesome dashboard.
>
> https://zephyr-dashboard.renode.io/
>
>
> It would be really cool if we could join forces a bit and continue to build
> off that effort and improve some of the emulation as needed (some work is
> required).


Wow, that is cool! I must have missed it when you mentioned it
previously. Yes, this is something the NuttX project should look into.

I agree with the QEMU idea as well, for those who want to build a
test/development setup they can run locally.

Both of these ideas are very good and we should pursue them. If we do
QEMU, we should document it, or script it, or both, so that other
community members can reproduce it and run a test system locally.
Personally I would like such a setup, and I am interested in helping
to document it.

Cheers,
Nathan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Alan C. Assis]

On 5/23/23, Brennan Ashton  wrote:
> On Tue, May 23, 2023, 5:05 AM Nathan Hartman 
> wrote:
>
>> On Tue, May 23, 2023 at 6:12 AM Brennan Ashton
>>  wrote:
>> > I have also asked in the past about cutting down on the amount of
>> > configs
>> > we have checked in to be something like
>> >
>> > board:nsh -- only nsh and somewhat small
>> > board:jumbo -- nsh, plus as many features as can fit and are
>> > interesting
>> in
>> > the platform.
>> >
>> > For sim and some other targets it would make sense to have more
>> > targets,
>> > but not for every board.
>>
>>
>> The idea of "board:jumbo" is very similar to what I was saying
>> earlier. Maybe it will allow us to test fewer boards in less time but
>> still get better test coverage. I am in favor of *better* test
>> coverage, not less test coverage!!
>>
>> In the past, we talked about having some tests in CI for each PR, and
>> then a bigger nightly test that builds all boards/configs like Greg
>> used to do before releases. I don't think that ever happened, but ASF
>> has a build farm separate from GitHub that we might use, or we could
>> request from INFRA a virtual machine to set up a complete environment.
>> Maybe that's something to think about.
>>
>
>
> I'm not sure why we would need anything new? We can still run this in
> GitHub actions, but generally I don't think we should be having PRs merge
> that are not passing build tests.
>
>
> As for more testing of system on boards, QEMU is great for some tests and
> there is a thin framework that does some of that work that Xiang and others
> have started.  A few years ago I also gave a talk to see if there was
> interest in working with the folks a renode.io. Their open source simulator
> is what Zypher is using and at the time had minimal support, but check out
> this awesome dashboard.
>
> https://zephyr-dashboard.renode.io/
>
>
> It would be really cool if we could join forces a bit and continue to build
> off that effort and improve some of the emulation as needed (some work is
> required).
>

Some time ago I contacted Michael (from Antmicro, Renode. I CC he
here) and he was very prone to help, but we didn't get a dashboard for
NuttX yet.

I totally agree with that idea, maybe we could create a page that will
show up at nuttx-dashboard.renode.io this way they don't need to
maintain it, since they already have much work with Zephyr.

> Nothing will beat hardware, but as Xiang said let's start with the easy bit
> which is software.
>

For sure! Let to start with QEMU/Renode and then will add real HW support.

BR,

Alan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Brennan Ashton]

On Tue, May 23, 2023, 5:05 AM Nathan Hartman 
wrote:

> On Tue, May 23, 2023 at 6:12 AM Brennan Ashton
>  wrote:
> > I have also asked in the past about cutting down on the amount of configs
> > we have checked in to be something like
> >
> > board:nsh -- only nsh and somewhat small
> > board:jumbo -- nsh, plus as many features as can fit and are interesting
> in
> > the platform.
> >
> > For sim and some other targets it would make sense to have more targets,
> > but not for every board.
>
>
> The idea of "board:jumbo" is very similar to what I was saying
> earlier. Maybe it will allow us to test fewer boards in less time but
> still get better test coverage. I am in favor of *better* test
> coverage, not less test coverage!!
>
> In the past, we talked about having some tests in CI for each PR, and
> then a bigger nightly test that builds all boards/configs like Greg
> used to do before releases. I don't think that ever happened, but ASF
> has a build farm separate from GitHub that we might use, or we could
> request from INFRA a virtual machine to set up a complete environment.
> Maybe that's something to think about.
>


I'm not sure why we would need anything new? We can still run this in
GitHub actions, but generally I don't think we should be having PRs merge
that are not passing build tests.


As for more testing of system on boards, QEMU is great for some tests and
there is a thin framework that does some of that work that Xiang and others
have started.  A few years ago I also gave a talk to see if there was
interest in working with the folks a renode.io. Their open source simulator
is what Zypher is using and at the time had minimal support, but check out
this awesome dashboard.

https://zephyr-dashboard.renode.io/


It would be really cool if we could join forces a bit and continue to build
off that effort and improve some of the emulation as needed (some work is
required).

Nothing will beat hardware, but as Xiang said let's start with the easy bit
which is software.

--Brennan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Tomek CEDRO]

On Tue, May 23, 2023 at 7:48 AM Xiang Xiao wrote:
> Why not start the test infrastructure from sim/qemu? It's more simple to
> set up and has unlimited resources. Once the sim/qemu test workflow is
> ready, it isn't very hard to duplicate to the real boards.

Yeah, *:nsh should work that way, and should allow some runtime tests on
various architectures that QEMU already provide :-)

https://www.qemu.org/docs/master/about/emulation.html

Emulation 

QEMU’s Tiny Code Generator (TCG) provides the ability to emulate a number
of CPU architectures on any supported host platform. Both System Emulation
 and User
Mode Emulation
 are
supported depending on the guest architecture.
Supported Guest Architectures for Emulation


Architecture (qemu name)

System

User

Notes

Alpha

Yes

Yes

Legacy 64 bit RISC ISA developed by DEC

Arm (arm, aarch64)

Yes


Yes

Wide range of features, see A-profile CPU architecture support

for details

AVR

Yes


No

8 bit micro controller, often used in maker projects

Cris

Yes

Yes

Embedded RISC chip developed by AXIS

Hexagon

No

Yes

Family of DSPs by Qualcomm

PA-RISC (hppa)

Yes

Yes

A legacy RISC system used in HP’s old minicomputers

x86 (i386, x86_64)

Yes


Yes

The ubiquitous desktop PC CPU architecture, 32 and 64 bit.

Loongarch

Yes

Yes

A MIPS-like 64bit RISC architecture developed in China

m68k

Yes


Yes

Motorola 68000 variants and ColdFire

Microblaze

Yes

Yes

RISC based soft-core by Xilinx

MIPS (mips*)

Yes


Yes

Venerable RISC architecture originally out of Stanford University

Nios2

Yes

Yes

32 bit embedded soft-core by Altera

OpenRISC

Yes


Yes

Open source RISC architecture developed by the OpenRISC community

Power (ppc, ppc64)

Yes


Yes

A general purpose RISC architecture now managed by IBM

RISC-V

Yes


Yes

An open standard RISC ISA maintained by RISC-V International

RX

Yes


No

A 32 bit micro controller developed by Renesas

s390x

Yes


Yes

A 64 bit CPU found in IBM’s System Z mainframes

sh4

Yes

Yes

A 32 bit RISC embedded CPU developed by Hitachi

SPARC (sparc, sparc64)

Yes


Yes

A RISC ISA originally developed by Sun Microsystems

Tricore

Yes

No

A 32 bit RISC/uController/DSP developed by Infineon

Xtensa

Yes


Yes

A configurable 32 bit soft core now owned by Cadence

A number of features are only available when running under emulation
including Record/Replay
 and QEMU TCG
Plugins
.
Semihosting


Semihosting is a feature defined by the owner of the architecture to allow
programs to interact with a debugging host system. On real hardware this is
usually provided by an In-circuit emulator (ICE) hooked directly to the
board. QEMU’s implementation allows for semihosting calls to be passed to
the host system or via the gdbstub.

Generally semihosting makes it easier to bring up low level code before a
more fully functional operating system has been enabled. On QEMU it also
allows for embedded micro-controller code which typically doesn’t have a
full libc to be run as “bare-metal” code under QEMU’s user-mode emulation.
It is also useful for writing test cases and indeed a number of compiler
suites as well as QEMU itself use semihosting calls to exit test code while
reporting the success state.

Semihosting is only available using TCG emulation. This is because the
instructions to trigger a semihosting call are typically reserved causing
most hypervisors to trap and fault on them.

Espressif adds their MCU support to qemu, they just added ESP32-C3 (RISC-V)

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Nathan Hartman]

On Tue, May 23, 2023 at 6:12 AM Brennan Ashton
 wrote:
> I have also asked in the past about cutting down on the amount of configs
> we have checked in to be something like
>
> board:nsh -- only nsh and somewhat small
> board:jumbo -- nsh, plus as many features as can fit and are interesting in
> the platform.
>
> For sim and some other targets it would make sense to have more targets,
> but not for every board.


The idea of "board:jumbo" is very similar to what I was saying
earlier. Maybe it will allow us to test fewer boards in less time but
still get better test coverage. I am in favor of *better* test
coverage, not less test coverage!!

In the past, we talked about having some tests in CI for each PR, and
then a bigger nightly test that builds all boards/configs like Greg
used to do before releases. I don't think that ever happened, but ASF
has a build farm separate from GitHub that we might use, or we could
request from INFRA a virtual machine to set up a complete environment.
Maybe that's something to think about.

Cheers,
Nathan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Brennan Ashton]

On Mon, May 22, 2023, 12:14 PM Nathan Hartman 
wrote:

> On Mon, May 22, 2023 at 9:29 AM Sebastien Lorquet 
> wrote:
>
> >
> > If the untold reason is to speed up github tests, then run less tests.
> > Do we really need to test build on 13 or 20 arm platforms when only one
> > config of the other architectures is tested, and the actual value of
> > these build test is dubious?



My grumbles about build times it not about CI (although I would love for
the steps like clean to not be so slow).  I would be quite opposed to
letting PRs through without passing current build tests as in the past
anything touching common code has broken builds in unexpected ways leaving
other people to clean the mess up.

We already have ways to limit what builds are run based on the files
changed so if we wanted to skip builds in the cases that only certain
board/arch changes are a touched that is possible with someone willing to
do the work.

You will note if you change only files in Documentation/** only the docs
get built.

I have also asked in the past about cutting down on the amount of configs
we have checked in to be something like

board:nsh -- only nsh and somewhat small
board:jumbo -- nsh, plus as many features as can fit and are interesting in
the platform.

For sim and some other targets it would make sense to have more targets,
but not for every board.

--Brennan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Lee, Lup Yuen]

Maybe we can run the CI Tests once a day, so it's easier to backtrack? FYI
I've run Automated Tests on BL602 NuttX over the past 365 days (except for
the brief Makefile outage last week):

https://github.com/lupyuen/nuttx/tags

Here's how it works:

https://lupyuen.github.io/articles/auto

Lup

On Tue, May 23, 2023 at 3:33 PM Sebastien Lorquet 
wrote:

> Hello,
>
> even if theoretically nice to do, do we really, actually, need to do
> that for the purpose of checking *every* pull request, which are quite
> numerous?
>
> Could that not be done once before a release?
>
> Sebastien
>
> Le 22/05/2023 à 22:31, Maciej Wójcik a écrit :
> > Checking different configurations is an academic problem, I think they
> call
> > it configuration sampling and it is part of variability modelling. There
> > were some papers about sampling of Linux configurations.
> >
> > The simplest approach is to enable all possible, disable all possible,
> but
> > it is not trivial. Each selection multiplies the number of configurations
> > by the number of available options. That has very bad complexity.
> >
> > They use SAT solvers to generate many configurations instead of brute
> > force. The goal is to sample configuration space in a uniform way.
> >
> > Am Mo., 22. Mai 2023 um 21:14 Uhr schrieb Nathan Hartman <
> > hartman.nat...@gmail.com>:
> >
> >> On Mon, May 22, 2023 at 9:29 AM Sebastien Lorquet  >
> >> wrote:
> >>
> >>> If the untold reason is to speed up github tests, then run less tests.
> >>> Do we really need to test build on 13 or 20 arm platforms when only one
> >>> config of the other architectures is tested, and the actual value of
> >>> these build test is dubious?
> >>
> >>
> >> This is an interesting point. It reminds me that (at least in the old
> days,
> >> I don't know now) FreeBSD had a build config that basically enabled all
> >> options, even if that's impossible for actually running, for build
> testing.
> >> I don't know if we can do that but maybe we need one ARM config that
> >> enables as many options as possible and then use other archs for other
> >> tests.
> >>
> >> Just a thought
> >>
> >> Nathan
> >>
>

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Sebastien Lorquet]

Hello,

even if theoretically nice to do, do we really, actually, need to do 
that for the purpose of checking *every* pull request, which are quite 
numerous?


Could that not be done once before a release?

Sebastien

Le 22/05/2023 à 22:31, Maciej Wójcik a écrit :

Checking different configurations is an academic problem, I think they call
it configuration sampling and it is part of variability modelling. There
were some papers about sampling of Linux configurations.

The simplest approach is to enable all possible, disable all possible, but
it is not trivial. Each selection multiplies the number of configurations
by the number of available options. That has very bad complexity.

They use SAT solvers to generate many configurations instead of brute
force. The goal is to sample configuration space in a uniform way.

Am Mo., 22. Mai 2023 um 21:14 Uhr schrieb Nathan Hartman <
hartman.nat...@gmail.com>:


On Mon, May 22, 2023 at 9:29 AM Sebastien Lorquet 
wrote:


If the untold reason is to speed up github tests, then run less tests.
Do we really need to test build on 13 or 20 arm platforms when only one
config of the other architectures is tested, and the actual value of
these build test is dubious?



This is an interesting point. It reminds me that (at least in the old days,
I don't know now) FreeBSD had a build config that basically enabled all
options, even if that's impossible for actually running, for build testing.
I don't know if we can do that but maybe we need one ARM config that
enables as many options as possible and then use other archs for other
tests.

Just a thought

Nathan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Xiang Xiao]

Why not start the test infrastructure from sim/qemu? It's more simple to
set up and has unlimited resources. Once the sim/qemu test workflow is
ready, it isn't very hard to duplicate to the real boards.

On Tue, May 23, 2023 at 8:42 AM Alan C. Assis  wrote:

> On 5/22/23, Tomek CEDRO  wrote:
> > This is why I asked not that long ago about
> > Software-Hardware-Support-Compatibility-Matrix.. this would be really
> > big table with hardware boards in columns and features in rows with
> > green marks (or +1) where full support is confirmed, yellow (or 0)
> > meaning work-in-progress, red (or -1) meaning no support or known
> > problems.
> >
> > According to that Compatibility Matrix it would be possible to create
> > proof-based configurations to build, and builds would prove the
> > configurations.
> >
> > To be honest I have no idea how that could be implemented in such a
> > complex project as NuttX with all those possible configurations.. that
> > would really require big CI automation and I am not really familiar
> > with GH CI yet maybe this is possible.. does GH charge $ for this CI
> > operations? :-)
> >
> > When working for ARM at mbed they had this big wall of boards and such
> > tests were performed not only at build stage but also on a real
> > hardware.. each board had DAPLink that allowed flashing and serial
> > port shell that executed some test scripts :-)
> >
>
> Yes, I and Sebastien tried to create a testing farm for NuttX using
> Raspberry Pi:
>
> https://bitbucket.org/acassis/raspi-nuttx-farm/src/master/
>
> but soon we realize it will not scale well, for each board we need a
> Raspi, or a USB HUB with Power Control over on each port (to
> physically turn ON/OFF the board).
>
> In the past using Raspberry Pi Zero was a good idea:
> https://www.raspberrypi.com/news/raspberry-pi-zero/
> The price was U$ 5.00, so we could by 100 that it was not too expensive :-)
>
> Maybe a better alternative should be create some USB/HUB board using
> ESP32-S2 that we could use as bridge to program from a central
> computer/board over WiFi.
>
> BR,
>
> Alan
>

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Alan C. Assis]

On 5/22/23, Tomek CEDRO  wrote:
> This is why I asked not that long ago about
> Software-Hardware-Support-Compatibility-Matrix.. this would be really
> big table with hardware boards in columns and features in rows with
> green marks (or +1) where full support is confirmed, yellow (or 0)
> meaning work-in-progress, red (or -1) meaning no support or known
> problems.
>
> According to that Compatibility Matrix it would be possible to create
> proof-based configurations to build, and builds would prove the
> configurations.
>
> To be honest I have no idea how that could be implemented in such a
> complex project as NuttX with all those possible configurations.. that
> would really require big CI automation and I am not really familiar
> with GH CI yet maybe this is possible.. does GH charge $ for this CI
> operations? :-)
>
> When working for ARM at mbed they had this big wall of boards and such
> tests were performed not only at build stage but also on a real
> hardware.. each board had DAPLink that allowed flashing and serial
> port shell that executed some test scripts :-)
>

Yes, I and Sebastien tried to create a testing farm for NuttX using
Raspberry Pi:

https://bitbucket.org/acassis/raspi-nuttx-farm/src/master/

but soon we realize it will not scale well, for each board we need a
Raspi, or a USB HUB with Power Control over on each port (to
physically turn ON/OFF the board).

In the past using Raspberry Pi Zero was a good idea:
https://www.raspberrypi.com/news/raspberry-pi-zero/
The price was U$ 5.00, so we could by 100 that it was not too expensive :-)

Maybe a better alternative should be create some USB/HUB board using
ESP32-S2 that we could use as bridge to program from a central
computer/board over WiFi.

BR,

Alan

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Tomek CEDRO]

This is why I asked not that long ago about
Software-Hardware-Support-Compatibility-Matrix.. this would be really
big table with hardware boards in columns and features in rows with
green marks (or +1) where full support is confirmed, yellow (or 0)
meaning work-in-progress, red (or -1) meaning no support or known
problems.

According to that Compatibility Matrix it would be possible to create
proof-based configurations to build, and builds would prove the
configurations.

To be honest I have no idea how that could be implemented in such a
complex project as NuttX with all those possible configurations.. that
would really require big CI automation and I am not really familiar
with GH CI yet maybe this is possible.. does GH charge $ for this CI
operations? :-)

When working for ARM at mbed they had this big wall of boards and such
tests were performed not only at build stage but also on a real
hardware.. each board had DAPLink that allowed flashing and serial
port shell that executed some test scripts :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: CI tests (was: Re: [OT] Learning Makefiles)

[Maciej Wójcik]

Checking different configurations is an academic problem, I think they call
it configuration sampling and it is part of variability modelling. There
were some papers about sampling of Linux configurations.

The simplest approach is to enable all possible, disable all possible, but
it is not trivial. Each selection multiplies the number of configurations
by the number of available options. That has very bad complexity.

They use SAT solvers to generate many configurations instead of brute
force. The goal is to sample configuration space in a uniform way.

Am Mo., 22. Mai 2023 um 21:14 Uhr schrieb Nathan Hartman <
hartman.nat...@gmail.com>:

> On Mon, May 22, 2023 at 9:29 AM Sebastien Lorquet 
> wrote:
>
> >
> > If the untold reason is to speed up github tests, then run less tests.
> > Do we really need to test build on 13 or 20 arm platforms when only one
> > config of the other architectures is tested, and the actual value of
> > these build test is dubious?
>
>
>
> This is an interesting point. It reminds me that (at least in the old days,
> I don't know now) FreeBSD had a build config that basically enabled all
> options, even if that's impossible for actually running, for build testing.
> I don't know if we can do that but maybe we need one ARM config that
> enables as many options as possible and then use other archs for other
> tests.
>
> Just a thought
>
> Nathan
>