[coreboot] Re: There is a python in our toolchain?!?

[David Hendricks]

As a rule of thumb, any project involving a substantial amount of Python
always ends up needing a Docker container to build. So I'm in the "no" camp
for making Python a dependency, however I think it's fine to keep things
as-is where it can be used for helper scripts and utilities for specific
purposes such that they aren't critical to building the tree.

On Wed, Sep 29, 2021 at 2:58 AM Patrick Georgi via coreboot <
coreboot@coreboot.org> wrote:

> That said, python makes its way back into the tree every now and then
> (typically as small snippets to compute and add hashes to binaries as
> needed by ARM SoCs). Uncanny, but typically not a big deal.
>
...
> To avoid these scenarios, could we possibly nail down the policy on python
> in coreboot?
>

The policy should be simple: The CI system (Jenkins) must be able to build
every target in its default configuration.

If we introduce Python as a dependency, then all Python in the tree must be
compatible with whatever version Jenkins uses. And if we're going to impose
the burden of fixing Python on everyone, then all developers must have the
ability to install a compatible version in their OS. Given the experiences
many of us in this thread have had and how widely distros vary in Python
support, I don't see this as tenable.

Another thing to keep in mind is that we have these sorts of helper scripts
from multiple vendors/parties over several years, and we'll likely see more
in the future. Pushing them all to use whatever version(s) of Python we
decide to build with does not seem realistic.

All that said, I'm fine with Python being used for helper scripts and such
as we've done in the past. It gives developers/vendors/etc. freedom to use
whatever works for their purposes without imposing a huge burden on
everyone else.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Peter Stuge]

Thanks for bringing this to the list, Patrick.

Stefan Reinauer wrote:
> Given the mess that Python 2 to Python 3 conversion has been (and
> still is), this is just inviting a lot of trouble into what has
> been a fairly stable part of coreboot for the last decade.

I strongly agree.

On a more general note I find it super sad that distributions purge
Python 2 only because it is EOL upstream when it still has many use
cases. It's not a problem for me but it does demonstrate how little
critical thinking happens within distributions.

So projects must be extra critical about dependencies.


I prefer changing nothing (change != progress) or banning Python
for the coreboot build. Utilities (and payloads) not needed for the
actual coreboot build can continue to use it.

openssl + dd could probably calculate and inject hashes.


Thanks

//Peter
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Matt B]

> What coreboot problems that we have seen in the past are we actually
solving with these rewrites?
To be a bit more blunt, what is expected to be improved by writing it in
python?
Utilities, eg. to analyse blobs?
Menu and configuration of builds?
Packing blobs for flashing?

I could see some things being harmless, like using python to drive menus
and check dependencies. I could potentially see a case that it's preferable
to complicated #ifdef logic and for handling of board variants.

On the other hand, I could definitely see python adding new ways that
reproducible builds fail, unless you compile a specific python interpreter
on the spot. Who knows what subtle differences there might be in python's
handling of binary data, especially on different arches and systems with
different endianness? Maybe I'm paranoid, or maybe this stuff could be
caught well enough in automated testing (and $someone could fix it).

Perhaps user-driven utils like those that extract blobs or examine MSRs to
discover interrupt routing on proprietary firmware are a safer middle
ground between the two extremes, but those are already written in C aren't
they?

>Python is already being used, in the "util" directory. util/qualcomm has
~3500 LoC in Python [1] (this is not just a "small snippet" of code).
How was it introduced? Was this from a vendor dump? Those seem to
occasionally bend the rules of what would be "ideal."

-Matt


On Wed, Sep 29, 2021 at 5:15 PM Ricardo Quesada via coreboot <
coreboot@coreboot.org> wrote:

> Thanks Patrick for bringing this in.
>
> Regarding Pytest for utils (https://review.coreboot.org/c/coreboot/+/57869
> ), what's the recommended way to test the tools that are placed in "util/*"
> ?
> I noticed that most of them don't have any kind of end-to-end test (I
> couldn't find a single one, but perhaps I missed them).
>
> Using PyTest (or any other easy to install / easy to use framework) to
> create tests for util/* seems natural to me.
> Although any other test framework can be used. Suggestions?
>
> Besides all the things that have been said in favor or against Python, I'd
> like to add that:
>Python is already being used, in the "util" directory. util/qualcomm
> has ~3500 LoC in Python [1] (this is not just a "small snippet" of code).
>(compare it with PyTests for elogutil[2]: It has ~160 LoC ).
>
> [1]: https://github.com/coreboot/coreboot/tree/master/util/qualcomm
> [2]: https://review.coreboot.org/c/coreboot/+/57869
>
>
> Thanks.
>
>
> On Wed, Sep 29, 2021 at 12:09 PM Stefan Reinauer <
> stefan.k.reina...@gmail.com> wrote:
>
>> On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal 
>> wrote:
>>
>>> Overall I think introducing Python to the build would provide net
>>> benefit, mainly from Kconfiglib, but could also find other good uses in e2e
>>> tests like Ricardo was working on. Most people's Linux distros ship with a
>>> Python interpreter too, so most developers would be unlikely to notice the
>>> extra dependency introduction.
>>>
>>> In terms of Kconfiglib, we have a lot to gain by switching away from the
>>> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
>>> forked from the Linux tree and hacked in our own customizations
>>> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
>>> a miniature Python script
>>> 
>>> that we use to handle our custom header format, and a stable API to work
>>> off of so that we can uprev Kconfiglib without needing to change our
>>> scripts.
>>>
>>
>> Looking at the current Kconfiglib implementation we would be replacing
>> the C code with 21873 lines of Python code that is now taking the code to
>> deviate from what the Linux kernel is doing. I am having a hard time seeing
>> a "net benefit" in this scenario. Given the mess that Python 2 to Python 3
>> conversion has been (and still is), this is just inviting a lot of trouble
>> into what has been a fairly stable part of coreboot for the last decade.
>>
>> In terms of Kconfiglib's stability and track record: I think it has it
>>> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
>>> without any issues at all.
>>>
>>
>> I am failing to see how anybody involved in coreboot would sign up for
>> and commit to porting 20k lines of Python code to the next version, when it
>> arrives. My indication is that not even the python code that is currently
>> in the tree has been ported to python3 yet.
>>
>>
>>> At a minimum, I think we should consider introducing Python on an
>>> optional basis (i.e., the C Kconfig implementation only gets used if a
>>> Python interpreter is unavailable), but making it required would be even
>>> better.
>>>
>>
>> Come on, we do not need two types of Kconfig parsers in the tree. Let's
>> focus coreboot on actually booting cores, not collections of
>> implementations in different programming languages. There are better

[coreboot] Re: There is a python in our toolchain?!?

[Ricardo Quesada via coreboot]

Thanks Patrick for bringing this in.

Regarding Pytest for utils (https://review.coreboot.org/c/coreboot/+/57869
), what's the recommended way to test the tools that are placed in "util/*"
?
I noticed that most of them don't have any kind of end-to-end test (I
couldn't find a single one, but perhaps I missed them).

Using PyTest (or any other easy to install / easy to use framework) to
create tests for util/* seems natural to me.
Although any other test framework can be used. Suggestions?

Besides all the things that have been said in favor or against Python, I'd
like to add that:
   Python is already being used, in the "util" directory. util/qualcomm has
~3500 LoC in Python [1] (this is not just a "small snippet" of code).
   (compare it with PyTests for elogutil[2]: It has ~160 LoC ).

[1]: https://github.com/coreboot/coreboot/tree/master/util/qualcomm
[2]: https://review.coreboot.org/c/coreboot/+/57869


Thanks.


On Wed, Sep 29, 2021 at 12:09 PM Stefan Reinauer <
stefan.k.reina...@gmail.com> wrote:

> On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal 
> wrote:
>
>> Overall I think introducing Python to the build would provide net
>> benefit, mainly from Kconfiglib, but could also find other good uses in e2e
>> tests like Ricardo was working on. Most people's Linux distros ship with a
>> Python interpreter too, so most developers would be unlikely to notice the
>> extra dependency introduction.
>>
>> In terms of Kconfiglib, we have a lot to gain by switching away from the
>> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
>> forked from the Linux tree and hacked in our own customizations
>> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
>> a miniature Python script
>> 
>> that we use to handle our custom header format, and a stable API to work
>> off of so that we can uprev Kconfiglib without needing to change our
>> scripts.
>>
>
> Looking at the current Kconfiglib implementation we would be replacing the
> C code with 21873 lines of Python code that is now taking the code to
> deviate from what the Linux kernel is doing. I am having a hard time seeing
> a "net benefit" in this scenario. Given the mess that Python 2 to Python 3
> conversion has been (and still is), this is just inviting a lot of trouble
> into what has been a fairly stable part of coreboot for the last decade.
>
> In terms of Kconfiglib's stability and track record: I think it has it
>> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
>> without any issues at all.
>>
>
> I am failing to see how anybody involved in coreboot would sign up for and
> commit to porting 20k lines of Python code to the next version, when it
> arrives. My indication is that not even the python code that is currently
> in the tree has been ported to python3 yet.
>
>
>> At a minimum, I think we should consider introducing Python on an
>> optional basis (i.e., the C Kconfig implementation only gets used if a
>> Python interpreter is unavailable), but making it required would be even
>> better.
>>
>
> Come on, we do not need two types of Kconfig parsers in the tree. Let's
> focus coreboot on actually booting cores, not collections of
> implementations in different programming languages. There are better
> projects for that, such as https://github.com/mame/quine-relay
>
> What coreboot problems that we have seen in the past are we actually
> solving with these rewrites?
>
> Stefan
>
>
>>
>> On Wed, Sep 29, 2021 at 5:39 AM Rao G  wrote:
>>
>>> Hi Patrick,
>>>
>>> That's good to hear, would there be change to "make menuconfig" with
>>> kconfiglib
>>>
>>
>> No, we'd make it so that all the "make *config" commands run the
>> Kconfiglib alternatives without any user action required.
>>
>> The "menuconfig" interface in specific is very similar to the
>> lxdialog-based interface that the C Kconfig uses, except it's a bit more
>> polished and refined feeling.
>>
>>
>>>
>>> Thanks
>>> Ranga
>>>
>>> On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <
>>> coreboot@coreboot.org> wrote:
>>>
 Hi everybody,

 Historically, coreboot avoided depending on python too much (we got rid
 of an entire python based configuration and build system, for example),
 with few minor exceptions.

 The main reason has been that while python code is quick to slap
 together, it has demonstrated a penchant for breaking in all kinds of
 mysterious ways (python 2->3 really was just a slightly bigger instance of
 what's going on in python all the time), and its users demonstrate a
 disregard for their fellow developers as demonstrated by endless stack
 traces on trivial errors (or is the language too complicated to properly
 catch them all?)

 While probably nice for one-off prototypes, long term maintenance is a
 concern: this project has over 20 

[coreboot] Probook hardware info

[Brian Milliron]

Since it didn't look like I would be able to get coreboot working on
this laptop, I returned it. I still have all the hardware info I
collected. Is this useful to anyone?
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Stefan Reinauer]

On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal  wrote:

> Overall I think introducing Python to the build would provide net benefit,
> mainly from Kconfiglib, but could also find other good uses in e2e tests
> like Ricardo was working on. Most people's Linux distros ship with a Python
> interpreter too, so most developers would be unlikely to notice the extra
> dependency introduction.
>
> In terms of Kconfiglib, we have a lot to gain by switching away from the
> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
> forked from the Linux tree and hacked in our own customizations
> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
> a miniature Python script
> 
> that we use to handle our custom header format, and a stable API to work
> off of so that we can uprev Kconfiglib without needing to change our
> scripts.
>

Looking at the current Kconfiglib implementation we would be replacing the
C code with 21873 lines of Python code that is now taking the code to
deviate from what the Linux kernel is doing. I am having a hard time seeing
a "net benefit" in this scenario. Given the mess that Python 2 to Python 3
conversion has been (and still is), this is just inviting a lot of trouble
into what has been a fairly stable part of coreboot for the last decade.

In terms of Kconfiglib's stability and track record: I think it has it
> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
> without any issues at all.
>

I am failing to see how anybody involved in coreboot would sign up for and
commit to porting 20k lines of Python code to the next version, when it
arrives. My indication is that not even the python code that is currently
in the tree has been ported to python3 yet.


> At a minimum, I think we should consider introducing Python on an optional
> basis (i.e., the C Kconfig implementation only gets used if a Python
> interpreter is unavailable), but making it required would be even better.
>

Come on, we do not need two types of Kconfig parsers in the tree. Let's
focus coreboot on actually booting cores, not collections of
implementations in different programming languages. There are better
projects for that, such as https://github.com/mame/quine-relay

What coreboot problems that we have seen in the past are we actually
solving with these rewrites?

Stefan


>
> On Wed, Sep 29, 2021 at 5:39 AM Rao G  wrote:
>
>> Hi Patrick,
>>
>> That's good to hear, would there be change to "make menuconfig" with
>> kconfiglib
>>
>
> No, we'd make it so that all the "make *config" commands run the
> Kconfiglib alternatives without any user action required.
>
> The "menuconfig" interface in specific is very similar to the
> lxdialog-based interface that the C Kconfig uses, except it's a bit more
> polished and refined feeling.
>
>
>>
>> Thanks
>> Ranga
>>
>> On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <
>> coreboot@coreboot.org> wrote:
>>
>>> Hi everybody,
>>>
>>> Historically, coreboot avoided depending on python too much (we got rid
>>> of an entire python based configuration and build system, for example),
>>> with few minor exceptions.
>>>
>>> The main reason has been that while python code is quick to slap
>>> together, it has demonstrated a penchant for breaking in all kinds of
>>> mysterious ways (python 2->3 really was just a slightly bigger instance of
>>> what's going on in python all the time), and its users demonstrate a
>>> disregard for their fellow developers as demonstrated by endless stack
>>> traces on trivial errors (or is the language too complicated to properly
>>> catch them all?)
>>>
>>> While probably nice for one-off prototypes, long term maintenance is a
>>> concern: this project has over 20 years of history under its belt, with
>>> more to come.
>>>
>>> That said, python makes its way back into the tree every now and then
>>> (typically as small snippets to compute and add hashes to binaries as
>>> needed by ARM SoCs). Uncanny, but typically not a big deal.
>>>
>>> There are two bigger initiatives proposed that would significantly
>>> increase our python footprint:
>>> 1. Replacing Linux's kconfig with kconfiglib (
>>> https://review.coreboot.org/c/coreboot/+/48679)
>>> 2. Using pytest for end-to-end testing utilities (
>>> https://review.coreboot.org/c/coreboot/+/57869)
>>>
>>> Compared to the "inject a hash value at a fixed location" scripts, these
>>> would probably be here to stay, and sufficiently integrated that everybody
>>> will have to deal with them.
>>>
>>> People spending time working on python code when it has no chance to
>>> land isn't a good use of their time and we should avoid that in the project.
>>>
>>> People spending time arguing that python shouldn't be used (to avoid
>>> the other outcome) even though the project's culture shifted and is now
>>> accepting Python isn't creating a 

[coreboot] Re: There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Am Mi., 29. Sept. 2021 um 19:47 Uhr schrieb Jack Rosenthal <
jrose...@google.com>:

> At a minimum, I think we should consider introducing Python on an optional
>>> basis (i.e., the C Kconfig implementation only gets used if a Python
>>> interpreter is unavailable), but making it required would be even better.
>>>
>> That idea is... horrible. Instead of only bringing in the python
>> dependency we'd also create an environmental dependency that can silently
>> introduce behavioral differences.
>>
> Fair, require python then. Most people have it on their systems anyways.
>
Well, the question I put up on the mailing list, for the community to
decide, is if we should lift our quasi-ban on python in the project. I
added you and the other contributors proposing python changes in Cc to this
email as a courtesy because you're not all registered on the project's
discussion venue.

The default alternative to "let's do something horrible" isn't "let's do
this other thing instead", it's "don't change anything at all". Don't look
further than the requirement for proposed changes to make it through
review: Without "don't change anything" as default, we could just open up
write access to the main repo and branch for everybody and everything - we
don't.

Only if we decide that extensive python use in the coreboot tree is now
okay, kconfiglib is even eligible for discussion.

So far the only argument in favor of python in _any_ way has been "it's
installed everywhere". That was true 20 years ago as well, and yet still we
had a fair amount of trouble with python. Look no further than the recent
uptick in support requests on IRC by people trying to build coreboot +
"stable" Tianocore: It fails because python moved on while the tagged
Tianocore version did not and its build tooling relies on python. So I
agree that "installed everywhere" is a necessary condition, but given how
badly python's maintainers historically treated their user base, I don't
think it's a sufficient condition, by far.

tl;dr: people of coreboot, opinions?


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

On Wed, Sep 29, 2021 at 9:57 AM Patrick Georgi  wrote:

> Am Mi., 29. Sept. 2021 um 16:43 Uhr schrieb Jack Rosenthal <
> jrose...@google.com>:
>
>> Overall I think introducing Python to the build would provide net
>> benefit, mainly from Kconfiglib, but could also find other good uses in e2e
>> tests like Ricardo was working on. Most people's Linux distros ship with a
>> Python interpreter too, so most developers would be unlikely to notice the
>> extra dependency introduction.
>>
> This is assuming that all python environments are alike. Experience
> disagrees.
>

Kconfiglib only uses Python standard libraries, and works with both Python
3.3+ and 2.7. I think it'd be pretty hard to find a Python environment it
didn't work with.


>
>
>> In terms of Kconfiglib, we have a lot to gain by switching away from the
>> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
>> forked from the Linux tree and hacked in our own customizations
>> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
>> a miniature Python script
>> 
>> that we use to handle our custom header format, and a stable API to work
>> off of so that we can uprev Kconfiglib without needing to change our
>> scripts.
>>
> The customizations are a set of patches with simple maintenance (as
> documented in https://review.coreboot.org/c/coreboot/+/57879).
>

This is true, but the maintenance burden could be even less.

Additionally, the last Kconfig uprev didn't go so well, at least from the
Chrome OS perspective, we saw a build flake that sat around for a few weeks
bothering people.


>
> In terms of Kconfiglib's stability and track record: I think it has it
>> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
>> without any issues at all.
>>
> This project deals in 20 year timelines. Zephyr is at 5.5 years, while
> depthcharge is used exclusively in a tightly controlled environment (and
> even there we had - and have - our share of pythonic problems).
>

IMO 5.5 years of stability is a pretty good point to start considering
adopting a new technology. If we wait for 20 years of it, there'll be
something better than Kconfiglib out there ;)


>
> At a minimum, I think we should consider introducing Python on an optional
>> basis (i.e., the C Kconfig implementation only gets used if a Python
>> interpreter is unavailable), but making it required would be even better.
>>
> That idea is... horrible. Instead of only bringing in the python
> dependency we'd also create an environmental dependency that can silently
> introduce behavioral differences.
>
> This project has the ability of building bit-identical firmware images on
> hosts spanning all kinds of CPU register sizes, endianness and operating
> system families. If you ever heard of "hermetic builds" being a good idea:
> the trustworthiness of our build is stronger - and part of making that
> possible is being (relatively) careful with picking our dependencies.
>

Fair, require python then. Most people have it on their systems anyways.


>
>
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>


-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

Overall I think introducing Python to the build would provide net benefit,
mainly from Kconfiglib, but could also find other good uses in e2e tests
like Ricardo was working on. Most people's Linux distros ship with a Python
interpreter too, so most developers would be unlikely to notice the extra
dependency introduction.

In terms of Kconfiglib, we have a lot to gain by switching away from the
Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
forked from the Linux tree and hacked in our own customizations
(KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
a miniature Python script

that we use to handle our custom header format, and a stable API to work
off of so that we can uprev Kconfiglib without needing to change our
scripts.

In terms of Kconfiglib's stability and track record: I think it has it
covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
without any issues at all.

At a minimum, I think we should consider introducing Python on an optional
basis (i.e., the C Kconfig implementation only gets used if a Python
interpreter is unavailable), but making it required would be even better.

On Wed, Sep 29, 2021 at 5:39 AM Rao G  wrote:

> Hi Patrick,
>
> That's good to hear, would there be change to "make menuconfig" with
> kconfiglib
>

No, we'd make it so that all the "make *config" commands run the Kconfiglib
alternatives without any user action required.

The "menuconfig" interface in specific is very similar to the
lxdialog-based interface that the C Kconfig uses, except it's a bit more
polished and refined feeling.


>
> Thanks
> Ranga
>
> On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <
> coreboot@coreboot.org> wrote:
>
>> Hi everybody,
>>
>> Historically, coreboot avoided depending on python too much (we got rid
>> of an entire python based configuration and build system, for example),
>> with few minor exceptions.
>>
>> The main reason has been that while python code is quick to slap
>> together, it has demonstrated a penchant for breaking in all kinds of
>> mysterious ways (python 2->3 really was just a slightly bigger instance of
>> what's going on in python all the time), and its users demonstrate a
>> disregard for their fellow developers as demonstrated by endless stack
>> traces on trivial errors (or is the language too complicated to properly
>> catch them all?)
>>
>> While probably nice for one-off prototypes, long term maintenance is a
>> concern: this project has over 20 years of history under its belt, with
>> more to come.
>>
>> That said, python makes its way back into the tree every now and then
>> (typically as small snippets to compute and add hashes to binaries as
>> needed by ARM SoCs). Uncanny, but typically not a big deal.
>>
>> There are two bigger initiatives proposed that would significantly
>> increase our python footprint:
>> 1. Replacing Linux's kconfig with kconfiglib (
>> https://review.coreboot.org/c/coreboot/+/48679)
>> 2. Using pytest for end-to-end testing utilities (
>> https://review.coreboot.org/c/coreboot/+/57869)
>>
>> Compared to the "inject a hash value at a fixed location" scripts, these
>> would probably be here to stay, and sufficiently integrated that everybody
>> will have to deal with them.
>>
>> People spending time working on python code when it has no chance to land
>> isn't a good use of their time and we should avoid that in the project.
>>
>> People spending time arguing that python shouldn't be used (to avoid
>> the other outcome) even though the project's culture shifted and is now
>> accepting Python isn't creating a great community for anybody.
>>
>> To avoid these scenarios, could we possibly nail down the policy on
>> python in coreboot?
>>
>>
>> Regards,
>> Patrick
>> --
>> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
>> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
>> Hamburg
>> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>> ___
>> coreboot mailing list -- coreboot@coreboot.org
>> To unsubscribe send an email to coreboot-le...@coreboot.org
>>
>

-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Am Mi., 29. Sept. 2021 um 16:43 Uhr schrieb Jack Rosenthal <
jrose...@google.com>:

> Overall I think introducing Python to the build would provide net benefit,
> mainly from Kconfiglib, but could also find other good uses in e2e tests
> like Ricardo was working on. Most people's Linux distros ship with a Python
> interpreter too, so most developers would be unlikely to notice the extra
> dependency introduction.
>
This is assuming that all python environments are alike. Experience
disagrees.


> In terms of Kconfiglib, we have a lot to gain by switching away from the
> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
> forked from the Linux tree and hacked in our own customizations
> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
> a miniature Python script
> 
> that we use to handle our custom header format, and a stable API to work
> off of so that we can uprev Kconfiglib without needing to change our
> scripts.
>
The customizations are a set of patches with simple maintenance (as
documented in https://review.coreboot.org/c/coreboot/+/57879).

In terms of Kconfiglib's stability and track record: I think it has it
> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
> without any issues at all.
>
This project deals in 20 year timelines. Zephyr is at 5.5 years, while
depthcharge is used exclusively in a tightly controlled environment (and
even there we had - and have - our share of pythonic problems).

At a minimum, I think we should consider introducing Python on an optional
> basis (i.e., the C Kconfig implementation only gets used if a Python
> interpreter is unavailable), but making it required would be even better.
>
That idea is... horrible. Instead of only bringing in the python dependency
we'd also create an environmental dependency that can silently introduce
behavioral differences.

This project has the ability of building bit-identical firmware images on
hosts spanning all kinds of CPU register sizes, endianness and operating
system families. If you ever heard of "hermetic builds" being a good idea:
the trustworthiness of our build is stronger - and part of making that
possible is being (relatively) careful with picking our dependencies.


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Rao G]

Hi Patrick,

That's good to hear, would there be change to "make menuconfig" with
kconfiglib

Thanks
Ranga

On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <
coreboot@coreboot.org> wrote:

> Hi everybody,
>
> Historically, coreboot avoided depending on python too much (we got rid of
> an entire python based configuration and build system, for example), with
> few minor exceptions.
>
> The main reason has been that while python code is quick to slap together,
> it has demonstrated a penchant for breaking in all kinds of mysterious ways
> (python 2->3 really was just a slightly bigger instance of what's going on
> in python all the time), and its users demonstrate a disregard for their
> fellow developers as demonstrated by endless stack traces on trivial errors
> (or is the language too complicated to properly catch them all?)
>
> While probably nice for one-off prototypes, long term maintenance is a
> concern: this project has over 20 years of history under its belt, with
> more to come.
>
> That said, python makes its way back into the tree every now and then
> (typically as small snippets to compute and add hashes to binaries as
> needed by ARM SoCs). Uncanny, but typically not a big deal.
>
> There are two bigger initiatives proposed that would significantly
> increase our python footprint:
> 1. Replacing Linux's kconfig with kconfiglib (
> https://review.coreboot.org/c/coreboot/+/48679)
> 2. Using pytest for end-to-end testing utilities (
> https://review.coreboot.org/c/coreboot/+/57869)
>
> Compared to the "inject a hash value at a fixed location" scripts, these
> would probably be here to stay, and sufficiently integrated that everybody
> will have to deal with them.
>
> People spending time working on python code when it has no chance to land
> isn't a good use of their time and we should avoid that in the project.
>
> People spending time arguing that python shouldn't be used (to avoid
> the other outcome) even though the project's culture shifted and is now
> accepting Python isn't creating a great community for anybody.
>
> To avoid these scenarios, could we possibly nail down the policy on python
> in coreboot?
>
>
> Regards,
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
> ___
> coreboot mailing list -- coreboot@coreboot.org
> To unsubscribe send an email to coreboot-le...@coreboot.org
>
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Hi everybody,

Historically, coreboot avoided depending on python too much (we got rid of
an entire python based configuration and build system, for example), with
few minor exceptions.

The main reason has been that while python code is quick to slap together,
it has demonstrated a penchant for breaking in all kinds of mysterious ways
(python 2->3 really was just a slightly bigger instance of what's going on
in python all the time), and its users demonstrate a disregard for their
fellow developers as demonstrated by endless stack traces on trivial errors
(or is the language too complicated to properly catch them all?)

While probably nice for one-off prototypes, long term maintenance is a
concern: this project has over 20 years of history under its belt, with
more to come.

That said, python makes its way back into the tree every now and then
(typically as small snippets to compute and add hashes to binaries as
needed by ARM SoCs). Uncanny, but typically not a big deal.

There are two bigger initiatives proposed that would significantly increase
our python footprint:
1. Replacing Linux's kconfig with kconfiglib (
https://review.coreboot.org/c/coreboot/+/48679)
2. Using pytest for end-to-end testing utilities (
https://review.coreboot.org/c/coreboot/+/57869)

Compared to the "inject a hash value at a fixed location" scripts, these
would probably be here to stay, and sufficiently integrated that everybody
will have to deal with them.

People spending time working on python code when it has no chance to land
isn't a good use of their time and we should avoid that in the project.

People spending time arguing that python shouldn't be used (to avoid
the other outcome) even though the project's culture shifted and is now
accepting Python isn't creating a great community for anybody.

To avoid these scenarios, could we possibly nail down the policy on python
in coreboot?


Regards,
Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org