I think there are two issues here. First, we can't get it all right at
once. But, second, we have to think in terms of eventual scale. The
system will need to make the status of hundreds of motherboards
accessible.
The Google doc is wonderful for a deep dive, but a bit overwhelming as
the first thing one sees.
I'm not sure the testportal scales that well either. The only report
system I've seen that approaches our needs is the chromeos build
waterfall, but even that may not be enough.
I think all the efforts to date were very good, and we need to learn
what we can from them. I also feel they are not quite right.
On Sat, Jul 17, 2021 at 2:45 AM Daniel Kulesz via coreboot
<coreboot@coreboot.org> wrote:
>
> Hi Patrick,
>
> > Looking at the software you described it seems a wonderful tool for humans
> > to create, execute test steps and analyse test results entered manually by
> > a human.
>
> Actually, this was the primary goal - we wanted to support testing of systems
> that are close to hardware (such as coreboot). Especially with coreboot,
> personally, I found too often that I bought a board that was officially
> "supported" just to find out that some things were actually broken while they
> seemed to have been working in past versions well (happened to me lately with
> a Thinkpad T410, see my previous postings to the list about this). The idea
> in SystemTestPortal is to support testing for such regressions - but this of
> course requires the availability of humans that execute these tests.
>
> > What we are looking for is only the "data store" and visual representation
> > of such, where automated tests are run by robots. Those (self-hosted or
> > propietary) robots need to publish their test results somewhere using a web
> > API.
>
> I see. Well, there is a different project named "ReportPortal.io" that (imho)
> does exactly that. We had a joint stand at FOSDEM 2019 together with them and
> KiwiTCMS. It might be worth looking into that.
>
> > Does SystemTestPortal support input from robots over for example REST API?
>
> Not at the moment but it is a feature request we received multiple times so
> we will eventually add this in the future.
>
> > Does it support the idea of different products/configurations?
>
> Yes, it does. We have a two-dimensional concept of a products and variants,
> but I think coreboot would need three dimensions or even more, right? So for
> example you could have:
>
> - coreboot 4.14 ("clean" without patches)
> - on a Thinkpad T410
> - built with config options X and Y enabled but Z disabled
>
> In addition, there could be also different configurations of the target
> machine, different OSs on which you would test etc. - the key challenge here
> is to decide what to put into the test cases themselves and what to put in
> the products/variants/configuration metadata. Maybe you could try to describe
> a data model that would be ideal from the coreboot perspective?
>
> Cheers, Daniel
> _______________________________________________
> coreboot mailing list -- coreboot@coreboot.org
> To unsubscribe send an email to coreboot-le...@coreboot.org
_______________________________________________
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org
