[coreboot] Re: The coreboot.org edk2 repo is live! (But don't get a heart attack)

2021-10-12 Thread Tim Crawford
> Finally, SMMSTORE: it exists, it helps where it is supported to persist
> UEFI variables, but as I understand it, actual support for devices is
> rather limited.

> SMMSTORE_V2 is fully working with the current coreboot default option
> for Tianocore (UEFIPAYLOAD), and is selected by default.

We are still using v1, but I want to switch us to v2 on our next edk2 update. 
My only concern is migrating the store from the v1 to the v2 format (or rather, 
the format edk2 writes when using v2). I was planning on doing it in our 
firmware-smmstore UEFI app, but if anyone else would need this functionality it 
would probably be better to add it somewhere in BlSMMStore.

I will rebase our changes on Matt's branch to see how many our fork still 
needs, probably some time next week.

-- 
Tim Crawford
System76
Kernel Engineer
tcrawf...@system76.com
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org


[coreboot] coreboot EFI working group meeting minutes - 12 October 2021

2021-10-12 Thread Martin Roth via coreboot
# 12 October 2021 - coreboot EFI working group

## Objective:

Linux is expecting more and more to use EFI supplied interfaces (UEFI
Boot Services in particular, even if many are stubbed out) so like it or
not, we’re going to need to support these interfaces. How do we approach
this without turning coreboot into UEFI.

## Decisions:

* The coreboot repo will host an EDK2 fork for use as a coreboot
  payload. Commits to this repo will go through gerrit for review.
* Any code needed to support booting should go into a payload instead of
  coreboot itself.
* We will look at adding chain-loading payload support to coreboot.
  Among other things, this will allow a payload to support turning cbmem
  tables into UEFI HOBs as needed before calling into a UEFI support
  payload.

## Minutes:

* Can you access the VC ? [Chris]
    * 9 people in. Apparently Chrome + Web or using the apps work for
  some people, but not for others.
    * Native [Bluejeans] apps
    * We’ll try a different method for our next meeting - The method
  will be announced after we decide when to hold our next meeting.
* FelixH: this isn’t a job for coreboot.
    * This should definitely be in a payload.
    * If FSP settings need to be configured, this would be a problem.
    * How would we support boot variables?  Or would we?
    * Werner: Agree with keeping this in a payload.
* [pgeorgi] What’s the scope of this project?
    * Just booting?
    * Jay: Syspro is typically asked to create solutions that are NOT
  UEFI.  They want simplicity and boot code to do the minimal
  amount.
    * 9elements: customers are mostly desiring UEFI boot solutions
  * coreboot need additional tables in cbmem to support a better
    UEFI implementation.
* What’s the timeline?
    * When will we NEED to support the EFI interfaces?
    * Do we know what these interfaces will be?
    * Initially just boot services.
* Is there any advantage to pulling EFI into coreboot.
    * No?  Having chainloading payloads could handle any translation
  necessary without tying that to a specific payload. (ie. coreboot
  loads the cbmem2hob payload, which does its thing, then cbmem2hob
  loads the actual payload)
    * Look into chain-loading payloads:  pgeorgi will look into this.
  Proof of concept: move splashscreen support out of ramstage.
* Add an edk2 fork to the coreboot git server
    * Where do we start from?
    * Current EDK2 stable from August 2021.
    * https://review.coreboot.org/edk2.git is now mirrored from
  https://github.com/tianocore/edk2
    * Updated hourly
    * Our own stable tag is default: coreboot-stable202108, based
  off edk2-stable202108
    * How do we pull changes from EDK2
    * Create upstream branch to pull in changes?  The vast majority
  of changes will come into this branch, and someone would need
  to sort them out at some point.
    * What’s the volume of patches that come into the EDK2?  That
  has a big impact to how we address this:
    * About 5 patches a day.
  * checked again: there are ~6000 commits between
    2020-09-20 and 2021-09-20 on edk2/master, so more like
    20 commits a day
    * Can we just create a fork and add things on top, or is there a
  better way to start?
    * Matt currently just creates his fork for the chromebook
  firmwares that he maintains.
    * How do we push changes back to EDK2?
    * This is a big issue - this is why many of the forks were
  created.
    * What’s the volume of our changes that would go into this tree?
    * How do we want to handle commits?
    * Are there enough people to do real code reviews?
    * Let’s just start by pushing through gerrit.
* What other more minimal projects are we interested in pulling in as
  payloads?
    * Yabits, Voodoo, U-Boot’s codebase?
    * Yabits did not work the last time it was (minimally tested)
    * Tianocore is currently not very well featured.
    * There are definitely situations currently where a uefi-payload
  is needed, so if there were something that provided a UEFI
  interface without all of the associated overhead, that would
  be very useful.
    * Everyone has different ideas about what’s needed as a minimal
  solution.
    * Voodoo is trying to get away from running all of this code in
  ring0.
    * No serious discussion about building anything new from scratch
  at this point, but that’s obviously not out of the question if
  someone wanted to start a new project.
* Definitely no one-size fits all solution here
* AI: Need to find out what linux is going to need.
    * Is this just what’s required for grub, or by linux itself - those
  are different things.
    * May be useful to have multiple implementations to find where the
  gaps are.
    * Any 

[coreboot] Re: The coreboot.org edk2 repo is live! (But don't get a heart attack)

2021-10-12 Thread Matt DeVillier
thanks for getting this set up Patrick!

On Tue, Oct 12, 2021 at 1:10 PM Patrick Georgi via coreboot
 wrote:
>
> Hi everybody,
>
> To facilitate cooperation on UEFI-as-a-payload work, we established a mirror 
> of tianocore's edk2 repo at https://review.coreboot.org/edk2. Unlike other 
> mirrors on review.coreboot.org, it's open for development.
>
> It's updated regularly, but the default branch that we set up, 
> coreboot-stable202108, is based on edk2-stable202108, so there won't be 
> changes flowing in automatically to the branch we will focus on.
>
> We will set up builders on qa.coreboot.org to cover that repo, so we get the 
> same "at the very least, it builds" guarantees that we have for any coreboot 
> contributions. Maybe we'll even get boot tests in the future, who knows?
>
> If you want to make coreboot+edk2 a viable option for starting hardware (with 
> the bonus compared to "regular" edk2 flows that hardware init happens on the 
> coreboot side, so if you want the same hardware to boot differently, it can 
> easily be made to be coreboot+SomethingElse!), there's plenty of 
> opportunities for developers.
>
> Matt DeVillier (Mr. Chromebox) offered to push his patch train there which 
> (as I understand it) is an amalgamation of changes made in various edk2 forks 
> in the coreboot ecosystem.
>
> Something people have talked about is adding Microsoft's Project Mu 
> (https://microsoft.github.io/mu/) UI improvements to tianocore-as-a-payload, 
> which could find a good home there, as well.
>
> Finally, SMMSTORE: it exists, it helps where it is supported to persist UEFI 
> variables, but as I understand it, actual support for devices is rather 
> limited.

SMMSTORE_V2 is fully working with the current coreboot default option
for Tianocore (UEFIPAYLOAD), and is selected by default. It should
work on all boards for which the default Tianocore option does;
currently it persists most (all?) EFI variables, including the boot
order entries. Most of the credit for this goes to the fine folks at
9elements.

> Making coreboot+edk2 the premier option for booting UEFI platforms would give 
> the rest of us something to work with that is more pleasant than trying to 
> NERF vendor firmware until it stops doing all the things we don't need it to 
> do.
>
> And if you don't care about UEFI (or if that's putting it mildly, even), 
> don't worry: this is only a payload. Just like we have FILO on our server or 
> SeaBIOS or depthcharge, this is just another option. But given the market 
> penetration of UEFI interfaces, it's an important one to get right.
>
>
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: 
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
> ___
> coreboot mailing list -- coreboot@coreboot.org
> To unsubscribe send an email to coreboot-le...@coreboot.org
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org


[coreboot] The coreboot.org edk2 repo is live! (But don't get a heart attack)

2021-10-12 Thread Patrick Georgi via coreboot
Hi everybody,

To facilitate cooperation on UEFI-as-a-payload work, we established a
mirror of tianocore's edk2 repo at https://review.coreboot.org/edk2. Unlike
other mirrors on review.coreboot.org, it's open for development.

It's updated regularly, but the default branch that we set up,
coreboot-stable202108, is based on edk2-stable202108, so there won't be
changes flowing in automatically to the branch we will focus on.

We will set up builders on qa.coreboot.org to cover that repo, so we get
the same "at the very least, it builds" guarantees that we have for any
coreboot contributions. Maybe we'll even get boot tests in the future, who
knows?

If you want to make coreboot+edk2 a viable option for starting hardware
(with the bonus compared to "regular" edk2 flows that hardware init happens
on the coreboot side, so if you want the same hardware to boot differently,
it can easily be made to be coreboot+SomethingElse!), there's plenty of
opportunities for developers.

Matt DeVillier (Mr. Chromebox) offered to push his patch train there which
(as I understand it) is an amalgamation of changes made in various edk2
forks in the coreboot ecosystem.

Something people have talked about is adding Microsoft's Project Mu (
https://microsoft.github.io/mu/) UI improvements to tianocore-as-a-payload,
which could find a good home there, as well.

Finally, SMMSTORE: it exists, it helps where it is supported to persist
UEFI variables, but as I understand it, actual support for devices is
rather limited.

Making coreboot+edk2 the premier option for booting UEFI platforms would
give the rest of us something to work with that is more pleasant than
trying to NERF vendor firmware until it stops doing all the things we don't
need it to do.

And if you don't care about UEFI (or if that's putting it mildly, even),
don't worry: this is only a payload. Just like we have FILO on our server
or SeaBIOS or depthcharge, this is just another option. But given the
market penetration of UEFI interfaces, it's an important one to get right.


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org


[coreboot] Re: There is a python in our toolchain?!?

2021-10-12 Thread ron minnich
contest is a coming standard. The companies I work with have moved away
from python test frameworks. Familiarity did not breed respect for python
test frameworks.

On Tue, Oct 12, 2021 at 7:31 AM Jack Rosenthal  wrote:

> Both bats and expect seem problematic for Ricardo's use case ...
> generating the elog binary format in these tools seems difficult, bash
> wasn't really meant for generating C structs.
>
> One huge advantage of pytest is that it's fairly industry standard at this
> point. There's a good number of people who have used it, and there's a
> large community around it (i.e., finding Q on sites like stack overflow
> is possible). I wasn't able to find a single stack overflow question about
> ConTest, although I may be looking in the wrong places (looks like they do
> have a slack group chat).
>
> I believe Ricardo had an open question on the CL if it was OK to submit as
> an optional test suite for elogtool. I.e., tests don't need to run on CI
> tools or anything, if there's developers that want to run the tests they
> can if they have Python.
>
>
>
> On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi  wrote:
>
>> Hi Ricardo,
>>
>> sorry for the late response, and that your project fell a bit by the
>> wayside. I guess discussion configuration frameworks is more attractive to
>> this community than testing frameworks (which also explains why we have ~3
>> config frameworks and only ~1 testing frameworks ;-) )
>>
>> So yes, testing is something we really need to improve on. I'm not sure
>> if "contest" is the right solution to your particular problem though. The
>> first thing I see when opening up its page is something about mysql, and
>> scrolling down, something about submitting jobs to a server. That seems
>> more like a potential replacement to our Jenkins install (qa.coreboot.org)
>> rather than something to easily write end-to-end tests for our userland
>> tools.
>>
>> Looking for options, my first instinct was to go for expect(1), but
>> that's really best for interactive uses - might be interesting if we ever
>> grow interactive tools, but so far we manage to stick to clean and tidy
>> CLI. Then I ran into https://github.com/bats-core/bats-core. That seems
>> maintained, reasonably minimal in its dependencies, it emits TAP which is
>> as good as JUnit in terms of Jenkins-integration (so we can have
>> qa.coreboot.org parse the results and give meaningful feedback on
>> review.coreboot.org), and it seems to be fairly widely used for similar
>> things to what you're doing, see
>> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it
>> points to many, many code examples, e.g.
>> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
>> which should cover the basic "call some command, see what it did" scenario
>> quite nicely.
>>
>> Of course in the end it's all a matter of taste, and that's why I opened
>> the can of worms again that is Python-use in coreboot land. As python
>> hasn't seen a warmer reception than last time, I'd look for alternatives.
>> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if
>> writing tests with that makes you want to scream and run away, we'd have to
>> look for other options (please, don't run away!)
>>
>>
>> Regards,
>> Patrick
>>
>> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada <
>> ricar...@google.com>:
>>
>>> Hi all,
>>>
>>> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend
>>> way to go forward?
>>> I just need to run one of the Coreboot's utils (in this case
>>> "elogtool"), and make sure the output is the expected one.
>>>
>>> Should I use Contest, as suggested by Ron Minnich?
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:
>>>
 Speaking as the person who wrote the first few config tools in python,
 and was happy to see the python dependency gone, I think bringing
 python back in would be a mistake.

 Every single python test framework I've worked with works until there
 is a problem, and I then find myself having to walk back a python call
 trace, because what inevitably breaks is the test framework tooling.
 It's why so many projects are removing python test frameworks.

 If you want a good test framework, get the linuxboot fork of
 facebook's contest github.com/linuxboot/contest, written in Go, in use
 at scale near you.

 It's easy to let the joy of building a build system overwhelm the
 actual goals of a project. coreboot is not about being a build system.
 It's easy to fall into the trap of creating an ever more complex
 system that is more than is needed.

 On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
  wrote:
 >
 > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
 jrose...@google.com>:
 >>
 >> With respect to Kconfig, we (at Google) encountered a lovely build
 flake after the 

[coreboot] Re: There is a python in our toolchain?!?

2021-10-12 Thread Jack Rosenthal via coreboot
Both bats and expect seem problematic for Ricardo's use case ... generating
the elog binary format in these tools seems difficult, bash wasn't really
meant for generating C structs.

One huge advantage of pytest is that it's fairly industry standard at this
point. There's a good number of people who have used it, and there's a
large community around it (i.e., finding Q on sites like stack overflow
is possible). I wasn't able to find a single stack overflow question about
ConTest, although I may be looking in the wrong places (looks like they do
have a slack group chat).

I believe Ricardo had an open question on the CL if it was OK to submit as
an optional test suite for elogtool. I.e., tests don't need to run on CI
tools or anything, if there's developers that want to run the tests they
can if they have Python.



On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi  wrote:

> Hi Ricardo,
>
> sorry for the late response, and that your project fell a bit by the
> wayside. I guess discussion configuration frameworks is more attractive to
> this community than testing frameworks (which also explains why we have ~3
> config frameworks and only ~1 testing frameworks ;-) )
>
> So yes, testing is something we really need to improve on. I'm not sure if
> "contest" is the right solution to your particular problem though. The
> first thing I see when opening up its page is something about mysql, and
> scrolling down, something about submitting jobs to a server. That seems
> more like a potential replacement to our Jenkins install (qa.coreboot.org)
> rather than something to easily write end-to-end tests for our userland
> tools.
>
> Looking for options, my first instinct was to go for expect(1), but that's
> really best for interactive uses - might be interesting if we ever grow
> interactive tools, but so far we manage to stick to clean and tidy CLI.
> Then I ran into https://github.com/bats-core/bats-core. That seems
> maintained, reasonably minimal in its dependencies, it emits TAP which is
> as good as JUnit in terms of Jenkins-integration (so we can have
> qa.coreboot.org parse the results and give meaningful feedback on
> review.coreboot.org), and it seems to be fairly widely used for similar
> things to what you're doing, see
> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it
> points to many, many code examples, e.g.
> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
> which should cover the basic "call some command, see what it did" scenario
> quite nicely.
>
> Of course in the end it's all a matter of taste, and that's why I opened
> the can of worms again that is Python-use in coreboot land. As python
> hasn't seen a warmer reception than last time, I'd look for alternatives.
> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if
> writing tests with that makes you want to scream and run away, we'd have to
> look for other options (please, don't run away!)
>
>
> Regards,
> Patrick
>
> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada <
> ricar...@google.com>:
>
>> Hi all,
>>
>> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend
>> way to go forward?
>> I just need to run one of the Coreboot's utils (in this case "elogtool"),
>> and make sure the output is the expected one.
>>
>> Should I use Contest, as suggested by Ron Minnich?
>>
>> Thanks!
>>
>>
>>
>> On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:
>>
>>> Speaking as the person who wrote the first few config tools in python,
>>> and was happy to see the python dependency gone, I think bringing
>>> python back in would be a mistake.
>>>
>>> Every single python test framework I've worked with works until there
>>> is a problem, and I then find myself having to walk back a python call
>>> trace, because what inevitably breaks is the test framework tooling.
>>> It's why so many projects are removing python test frameworks.
>>>
>>> If you want a good test framework, get the linuxboot fork of
>>> facebook's contest github.com/linuxboot/contest, written in Go, in use
>>> at scale near you.
>>>
>>> It's easy to let the joy of building a build system overwhelm the
>>> actual goals of a project. coreboot is not about being a build system.
>>> It's easy to fall into the trap of creating an ever more complex
>>> system that is more than is needed.
>>>
>>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
>>>  wrote:
>>> >
>>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
>>> jrose...@google.com>:
>>> >>
>>> >> With respect to Kconfig, we (at Google) encountered a lovely build
>>> flake after the Kconfig uprev last month in the coreboot tree that took a
>>> couple of weeks to sort out and resolve. Some sort of automated validation
>>> that the code is working could have possibly helped. Of course, the C
>>> implementation of Kconfig has no tests at all. Some tests is better than
>>> nothing.
>>> >
>>> >
>>> > Let me put the record 

[coreboot] Re: There is a python in our toolchain?!?

2021-10-12 Thread ron minnich
I set up a raspberry pi as a device under test with contest, and ran
the contest controller in a mode that needed no mysql, and it took
about 14 minutes.

I'd still recommend looking at contest. I've found it very easy to use
and I am very new to it.

On Tue, Oct 12, 2021 at 7:02 AM Patrick Georgi  wrote:
>
> Hi Ricardo,
>
> sorry for the late response, and that your project fell a bit by the wayside. 
> I guess discussion configuration frameworks is more attractive to this 
> community than testing frameworks (which also explains why we have ~3 config 
> frameworks and only ~1 testing frameworks ;-) )
>
> So yes, testing is something we really need to improve on. I'm not sure if 
> "contest" is the right solution to your particular problem though. The first 
> thing I see when opening up its page is something about mysql, and scrolling 
> down, something about submitting jobs to a server. That seems more like a 
> potential replacement to our Jenkins install (qa.coreboot.org) rather than 
> something to easily write end-to-end tests for our userland tools.
>
> Looking for options, my first instinct was to go for expect(1), but that's 
> really best for interactive uses - might be interesting if we ever grow 
> interactive tools, but so far we manage to stick to clean and tidy CLI. Then 
> I ran into https://github.com/bats-core/bats-core. That seems maintained, 
> reasonably minimal in its dependencies, it emits TAP which is as good as 
> JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse 
> the results and give meaningful feedback on review.coreboot.org), and it 
> seems to be fairly widely used for similar things to what you're doing, see 
> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points 
> to many, many code examples, e.g. 
> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
>  which should cover the basic "call some command, see what it did" scenario 
> quite nicely.
>
> Of course in the end it's all a matter of taste, and that's why I opened the 
> can of worms again that is Python-use in coreboot land. As python hasn't seen 
> a warmer reception than last time, I'd look for alternatives.
> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if 
> writing tests with that makes you want to scream and run away, we'd have to 
> look for other options (please, don't run away!)
>
>
> Regards,
> Patrick
>
> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada 
> :
>>
>> Hi all,
>>
>> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way 
>> to go forward?
>> I just need to run one of the Coreboot's utils (in this case "elogtool"), 
>> and make sure the output is the expected one.
>>
>> Should I use Contest, as suggested by Ron Minnich?
>>
>> Thanks!
>>
>>
>>
>> On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:
>>>
>>> Speaking as the person who wrote the first few config tools in python,
>>> and was happy to see the python dependency gone, I think bringing
>>> python back in would be a mistake.
>>>
>>> Every single python test framework I've worked with works until there
>>> is a problem, and I then find myself having to walk back a python call
>>> trace, because what inevitably breaks is the test framework tooling.
>>> It's why so many projects are removing python test frameworks.
>>>
>>> If you want a good test framework, get the linuxboot fork of
>>> facebook's contest github.com/linuxboot/contest, written in Go, in use
>>> at scale near you.
>>>
>>> It's easy to let the joy of building a build system overwhelm the
>>> actual goals of a project. coreboot is not about being a build system.
>>> It's easy to fall into the trap of creating an ever more complex
>>> system that is more than is needed.
>>>
>>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
>>>  wrote:
>>> >
>>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal 
>>> > :
>>> >>
>>> >> With respect to Kconfig, we (at Google) encountered a lovely build flake 
>>> >> after the Kconfig uprev last month in the coreboot tree that took a 
>>> >> couple of weeks to sort out and resolve. Some sort of automated 
>>> >> validation that the code is working could have possibly helped. Of 
>>> >> course, the C implementation of Kconfig has no tests at all. Some tests 
>>> >> is better than nothing.
>>> >
>>> >
>>> > Let me put the record straight:
>>> >
>>> > - The last kconfig "uprev" hasn't been a simple update the way 
>>> > https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire 
>>> > build system integration to ease maintenance
>>> > - That issue sprang up because before the kconfig update, we were 
>>> > shipping prebuilt parser files (C code) while now we made bison and flex 
>>> > hard dependencies for our build
>>> > - Tests covering the C code wouldn't have helped, because the issue 
>>> > wasn't in the C code
>>> > - The issue we were facing has been an external 

[coreboot] Re: There is a python in our toolchain?!?

2021-10-12 Thread Patrick Georgi via coreboot
Hi Ricardo,

sorry for the late response, and that your project fell a bit by the
wayside. I guess discussion configuration frameworks is more attractive to
this community than testing frameworks (which also explains why we have ~3
config frameworks and only ~1 testing frameworks ;-) )

So yes, testing is something we really need to improve on. I'm not sure if
"contest" is the right solution to your particular problem though. The
first thing I see when opening up its page is something about mysql, and
scrolling down, something about submitting jobs to a server. That seems
more like a potential replacement to our Jenkins install (qa.coreboot.org)
rather than something to easily write end-to-end tests for our userland
tools.

Looking for options, my first instinct was to go for expect(1), but that's
really best for interactive uses - might be interesting if we ever grow
interactive tools, but so far we manage to stick to clean and tidy CLI.
Then I ran into https://github.com/bats-core/bats-core. That seems
maintained, reasonably minimal in its dependencies, it emits TAP which is
as good as JUnit in terms of Jenkins-integration (so we can have
qa.coreboot.org parse the results and give meaningful feedback on
review.coreboot.org), and it seems to be fairly widely used for similar
things to what you're doing, see
https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points
to many, many code examples, e.g.
https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
which should cover the basic "call some command, see what it did" scenario
quite nicely.

Of course in the end it's all a matter of taste, and that's why I opened
the can of worms again that is Python-use in coreboot land. As python
hasn't seen a warmer reception than last time, I'd look for alternatives.
Maybe Bats could do? Of course, I haven't _actually_ used it yet and if
writing tests with that makes you want to scream and run away, we'd have to
look for other options (please, don't run away!)


Regards,
Patrick

Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada <
ricar...@google.com>:

> Hi all,
>
> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend
> way to go forward?
> I just need to run one of the Coreboot's utils (in this case "elogtool"),
> and make sure the output is the expected one.
>
> Should I use Contest, as suggested by Ron Minnich?
>
> Thanks!
>
>
>
> On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:
>
>> Speaking as the person who wrote the first few config tools in python,
>> and was happy to see the python dependency gone, I think bringing
>> python back in would be a mistake.
>>
>> Every single python test framework I've worked with works until there
>> is a problem, and I then find myself having to walk back a python call
>> trace, because what inevitably breaks is the test framework tooling.
>> It's why so many projects are removing python test frameworks.
>>
>> If you want a good test framework, get the linuxboot fork of
>> facebook's contest github.com/linuxboot/contest, written in Go, in use
>> at scale near you.
>>
>> It's easy to let the joy of building a build system overwhelm the
>> actual goals of a project. coreboot is not about being a build system.
>> It's easy to fall into the trap of creating an ever more complex
>> system that is more than is needed.
>>
>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
>>  wrote:
>> >
>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
>> jrose...@google.com>:
>> >>
>> >> With respect to Kconfig, we (at Google) encountered a lovely build
>> flake after the Kconfig uprev last month in the coreboot tree that took a
>> couple of weeks to sort out and resolve. Some sort of automated validation
>> that the code is working could have possibly helped. Of course, the C
>> implementation of Kconfig has no tests at all. Some tests is better than
>> nothing.
>> >
>> >
>> > Let me put the record straight:
>> >
>> > - The last kconfig "uprev" hasn't been a simple update the way
>> https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the
>> entire build system integration to ease maintenance
>> > - That issue sprang up because before the kconfig update, we were
>> shipping prebuilt parser files (C code) while now we made bison and flex
>> hard dependencies for our build
>> > - Tests covering the C code wouldn't have helped, because the issue
>> wasn't in the C code
>> > - The issue we were facing has been an external dependency (namely: the
>> Chromium OS development environment shipping a broken version of bison(1))
>> > - Fixing bison in CrOS wouldn't have helped any because we have to
>> assume that other users come with the same kind of broken bison tool
>> > - The fix has been to ship pre-built files again to remove an external
>> dependency
>> >
>> > The alternative that we did actually consider was to add support for
>> building bison and flex to util/crossgcc/buildgcc. For