Re: Risks of deterministic builds
* Jan Nieuwenhuizen [2021-04-14 11:27]: > Jean Louis writes: > > > * Jan Nieuwenhuizen [2021-04-08 16:43]: > >> Martin writes: > >> > >> > Maybe freedom in "free software" shouldn't require from the code to be > >> > open neither. Let's just blindly trust some saint developers who > >> > cannot even control their own binaries. Actually today we are closer > >> > and closer to that sad scenario like never before in the history, > >> > because in fact most of the open-source and GNU "free software" > >> > nowadays base on blackboxed binary seeds that cannot be verified by > >> > the users not even by the core developers. > >> > >> The bootstrappable project, GNU Mes and GNU Guix are working to fix that > >> > >> > >> https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/ > >> https://fosdem.org/2021/schedule/event/gnumes/ > > > > Jan, in regards to this, how to start with first stage, how are you > > entering it? > > Will you use for example Emacs and hexl-mode to enter bytes? > > I am not, I'm downloading the initial 357-byte binary from > 'bootstrap-seeds' > > > https://raw.githubusercontent.com/oriansj/bootstrap-seeds/master/POSIX/x86/hex0-seed Thank you, it becomes clearer. > and then run this script > > > https://github.com/oriansj/stage0-posix/blob/master/x86/mescc-tools-seed-kaem.kaem Thank you, now this below also makes sense somehow. ### # Phase-0 Build hex0 from bootstrapped binary # ### ../bootstrap-seeds/POSIX/x86/hex0-seed hex0_x86.hex0 hex0 # hex0 should have the exact same checksum as hex0-seed as they are both supposed # to be built from hex0_x86.hex0 and by definition must be identical How I understand it, seed is downloaded as a binary, but then the source code of hex0 is used to make a binary of hex0 again, for verification that seed was alright? I have to study the explanation in full and especially the first 2-3 stages. Jean
Re: Risks of deterministic builds
Jean Louis writes: > * Jan Nieuwenhuizen [2021-04-08 16:43]: >> Martin writes: >> >> > Maybe freedom in "free software" shouldn't require from the code to be >> > open neither. Let's just blindly trust some saint developers who >> > cannot even control their own binaries. Actually today we are closer >> > and closer to that sad scenario like never before in the history, >> > because in fact most of the open-source and GNU "free software" >> > nowadays base on blackboxed binary seeds that cannot be verified by >> > the users not even by the core developers. >> >> The bootstrappable project, GNU Mes and GNU Guix are working to fix that >> >> >> https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/ >> https://fosdem.org/2021/schedule/event/gnumes/ > > Jan, in regards to this, how to start with first stage, how are you > entering it? > Will you use for example Emacs and hexl-mode to enter bytes? I am not, I'm downloading the initial 357-byte binary from 'bootstrap-seeds' https://raw.githubusercontent.com/oriansj/bootstrap-seeds/master/POSIX/x86/hex0-seed and then run this script https://github.com/oriansj/stage0-posix/blob/master/x86/mescc-tools-seed-kaem.kaem See below for an overview and the next steps. Note that we are only in the 3rd iteration of removing binary seeds from the bootstrap; so in a way this project is just starting. A year ago, Ludovic Courtès already prototyped a step to build the first packages in the initial ramdisk, such as to remove dependencies on the Guix build daemon. Also note that major forces are going into "unbootstrappable" projects, that may even undo the small steps that we have just achieved. It has become much harder to bootstrap GCC from source then 10 years ago: we need unmaintained earlier versions of GCC to do so. Dependencies on Python, Rust are moving up the package graph. I choose not to believe that "nobody cares" about bootstrappability, i.e., binary-blob-free computing, I just think we need to raise awareness about it if we don't want to lose it. Greetings, Janneke The bootstrappable effort to bootstrap a full GNU/Linux system from source consists mainly of three projects: Stage0 by Jeremiah Orians and others, GNU Mes by Janneke and others, and the actual bootstrap packaging and deployment in GNU Guix: https://guix.gnu.org/en/manual/en/html_node/Bootstrapping.html#Bootstrapping Efforts are ongoing to implement such a bootstrap in other distributions. The reduced binary seed bootstrap was developed in GNU Guix in and is used in production https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/ https://guix.gnu.org/blog/2020/guix-further-reduces-bootstrap-seed-to-25 There have been several efforts to avoid using a package manager such as GNU Guix and most hove been abandoned. Possibly because it's fun to start such a thing, and pretty easy even to do the first steps without isolated builds, (cross-)build, or packaging support. The first working development version of 'wip-full-source-bootstrap' in Guix triggered a yet another effort like that, the live-bootstrap https://github.com/fosslinux/live-bootstrap The full source bootstrap is not in production; a development preview lives here https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/commencement.scm?h=wip-full-source-bootstrap The package graph for this bootstrap; the bottom of the guix package graph as generated by 'guix graph' looks like this https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/gcc-core-mesboot0-fsb-no-guile-unversioned.png?h=wip-talks which can be split into several stages https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/stage-0.png?h=wip-talks https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/stage-1.png?h=wip-talks https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/stage-2.png?h=wip-talks https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/stage-mes.png?h=wip-talks it may be easier to read that from the slides of my FOSDEM'21 talk https://fosdem.org/2021/schedule/event/gnumes/attachments/slides/4531/export/events/attachments/gnumes/slides/4531/gnu_mes_fosdem21.pdf You can see it all starts with 'bootstrap-seeds'. You can "build" that package (the seeds are just a download, of course) by (checking out the guix wip-full-source-bootstrap branch, building it) and, doing $ ./pre-inst-env guix build -e '(@@ (gnu packages commencement) bootstrap-seeds)' /gnu/store/sg0fxjd0pyfn5sm0m33227i3rz32l6z6-bootstrap-seeds-1.0.0 --8<---cut here---start->8--- (define m2-planet-boot ;; The initial bootstrap package: no binary inputs except for a 357 byte ;; binary seed: `x86/hex0-seed'. [..] (let* ((bootstrap-seeds (assoc-ref %build-inputs "bootstrap-seeds")) (mescc-tools-seed (assoc-ref %build-inputs
Re: Risks of deterministic builds
* Jan Nieuwenhuizen [2021-04-08 16:43]: > Martin writes: > > > Maybe freedom in "free software" shouldn't require from the code to be > > open neither. Let's just blindly trust some saint developers who > > cannot even control their own binaries. Actually today we are closer > > and closer to that sad scenario like never before in the history, > > because in fact most of the open-source and GNU "free software" > > nowadays base on blackboxed binary seeds that cannot be verified by > > the users not even by the core developers. > > The bootstrappable project, GNU Mes and GNU Guix are working to fix that > > > https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/ > https://fosdem.org/2021/schedule/event/gnumes/ Jan, in regards to this, how to start with first stage, how are you entering it? Will you use for example Emacs and hexl-mode to enter bytes? Jean
Re: Risks of deterministic builds
* Jan Nieuwenhuizen [2021-04-08 23:47]: > Jean Louis writes: > > > * Jan Nieuwenhuizen [2021-04-08 16:43]: > >> Martin writes: > > > > From https://bootstrapping.miraheze.org/wiki/Stage0 > > > >> Design > > > >> Stage0 starts with only 1 thing: > > > >> 1) A sub 500 byte hex monitor [How you create it is up to you; I like > >> toggling it in manually myself] > > > >> from that starting point, I have provided in easy to audit form > >> (direct mapping between Hex, its effective assembly and a C > >> implementation) a series of hex utilities that are required for basic > >> development work. These files are in the stage1 folder. > > > > How do you toggle it manually itself? > > I believ those are Jeremia Orians's words and I don't think we've > arrived at that stage yet. We started off by cutting corners to get > something that "works", and went from there: improving the bootstrap, > removing more binary seeds, removing shortcuts. > > Greetings, > Janneke It seem that your statements are not aligned and harmonized with Jeremia Orian's words, I am not understanding. We already said we have to trust people like you, but it looks like you that we are not yet there, and I did liste and watch to your video. Is the full source bootstrap possible or not? Only if answer is YES, what is the exact first step to do? If answer is NO, I will learn more about it. What I wish to see is the very first step. Jean
Re: Risks of deterministic builds
* Jan Nieuwenhuizen [2021-04-08 23:40]: > >> The bootstrappable project, GNU Mes and GNU Guix are working to fix that > >> > >> > >> https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/ > >> https://fosdem.org/2021/schedule/event/gnumes/ > > > > Janneke, that is probably most important step for GNU and free > > software in general. > > Thanks, we believe so too. > > > I have read it, and researching, but yet do not get how to start. I > > have downloaded stage0, is that the place to start? > > It's currently not very easy to do this all by hand. Easiest to see > what's happening would be to use Guix: the bootstrap is fully "scripted" > there by packages and their dependencies. If you look at:o I hope you may see the viewpoint that if it cannot be done by hand and if it is too obscure that it defeats itself by principle. In my opinion every user with average building skills should be able to do it. I was programming in machine language on 65C02 CPU back in time, and by using BASIC, I could peek and poke with memory and enter machine instructions, and program games and software for advertising this way. May I again ask, if possible, please define the very first step, or point to documentation that I can understand how to start with the very first step, so that this steap is clear and becomes transparent by all means. I have downloaded stage0-Release_0.4.0, is that the very first step? Is the file stage0_monitor.hex0 the very first step? Jean Take action in Free Software Foundation campaigns: https://www.fsf.org/campaigns Sign an open letter in support of Richard M. Stallman https://rms-support-letter.github.io/
Re: Risks of deterministic builds
Jean Louis writes: > * Jan Nieuwenhuizen [2021-04-08 16:43]: >> Martin writes: > > From https://bootstrapping.miraheze.org/wiki/Stage0 > >> Design > >> Stage0 starts with only 1 thing: > >> 1) A sub 500 byte hex monitor [How you create it is up to you; I like >> toggling it in manually myself] > >> from that starting point, I have provided in easy to audit form >> (direct mapping between Hex, its effective assembly and a C >> implementation) a series of hex utilities that are required for basic >> development work. These files are in the stage1 folder. > > How do you toggle it manually itself? I believ those are Jeremia Orians's words and I don't think we've arrived at that stage yet. We started off by cutting corners to get something that "works", and went from there: improving the bootstrap, removing more binary seeds, removing shortcuts. Greetings, Janneke -- Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
Re: Risks of deterministic builds
* Jan Nieuwenhuizen [2021-04-08 16:43]: > Martin writes: > > > Maybe freedom in "free software" shouldn't require from the code to be > > open neither. Let's just blindly trust some saint developers who > > cannot even control their own binaries. Actually today we are closer > > and closer to that sad scenario like never before in the history, > > because in fact most of the open-source and GNU "free software" > > nowadays base on blackboxed binary seeds that cannot be verified by > > the users not even by the core developers. > > The bootstrappable project, GNU Mes and GNU Guix are working to fix that > > > https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/ > https://fosdem.org/2021/schedule/event/gnumes/ >From https://bootstrapping.miraheze.org/wiki/Stage0 > Design > Stage0 starts with only 1 thing: > 1) A sub 500 byte hex monitor [How you create it is up to you; I like > toggling it in manually myself] > from that starting point, I have provided in easy to audit form > (direct mapping between Hex, its effective assembly and a C > implementation) a series of hex utilities that are required for basic > development work. These files are in the stage1 folder. How do you toggle it manually itself? Jean
Re: Risks of deterministic builds
Jean Louis writes: > * Jan Nieuwenhuizen [2021-04-08 16:43]: >> Martin writes: >> >> > Maybe freedom in "free software" shouldn't require from the code to be >> > open neither. Let's just blindly trust some saint developers who >> > cannot even control their own binaries. Actually today we are closer >> > and closer to that sad scenario like never before in the history, >> > because in fact most of the open-source and GNU "free software" >> > nowadays base on blackboxed binary seeds that cannot be verified by >> > the users not even by the core developers. >> >> The bootstrappable project, GNU Mes and GNU Guix are working to fix that >> >> >> https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/ >> https://fosdem.org/2021/schedule/event/gnumes/ > > Janneke, that is probably most important step for GNU and free > software in general. Thanks, we believe so too. > I have read it, and researching, but yet do not get how to start. I > have downloaded stage0, is that the place to start? It's currently not very easy to do this all by hand. Easiest to see what's happening would be to use Guix: the bootstrap is fully "scripted" there by packages and their dependencies. If you look at: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/commencement.scm?h=wip-full-source-bootstrap you can see it all starts with 'bootstrap-seeds'. You can "build" that package (the seeds are just a download, of course) by (checking out the guix wip-full-source-bootstrap branch, building it) and, doing $ ./pre-inst-env guix build -e '(@@ (gnu packages commencement) bootstrap-seeds)' /gnu/store/sg0fxjd0pyfn5sm0m33227i3rz32l6z6-bootstrap-seeds-1.0.0 and you'll see the 357-byte hex0-seed in there. This is used to build the 'm2-planet-boot' package which uses kaem scripts (any shell will do) to build hex0, hex1, M0, M1, hex2 and m2-planet: $ ./pre-inst-env guix build -e '(@@ (gnu packages commencement) m2-planet-boot)' /gnu/store/qxsd6zg910zjgwsm13c39hys2g931ij8-m2-planet-boot-1.7.0-31-g358b6cf then onto mes-boot $ ./pre-inst-env guix build -e '(@@ (gnu packages commencement) mes-boot)' /gnu/store/dfwkand25k6nsmrk85p8hsa1yq74y314-mes-boot-0.22-305-g2ab4c5c67 and so on. In fact, each time you build or install a package using Guix, these (bootstrap) dependencies are resolved and -if necessary- all built automagically. The first working version of 'wip-full-source-bootstrap' triggered the 'live-bootstrap' effort: https://github.com/fosslinux/live-bootstrap aiming to script/redo (change, improve, ...) this bootstrap outside of/without Guix. Greetings, Janneke -- Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
Re: Risks of deterministic builds
* Jan Nieuwenhuizen [2021-04-08 16:43]: > Martin writes: > > > Maybe freedom in "free software" shouldn't require from the code to be > > open neither. Let's just blindly trust some saint developers who > > cannot even control their own binaries. Actually today we are closer > > and closer to that sad scenario like never before in the history, > > because in fact most of the open-source and GNU "free software" > > nowadays base on blackboxed binary seeds that cannot be verified by > > the users not even by the core developers. > > The bootstrappable project, GNU Mes and GNU Guix are working to fix that > > > https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/ > https://fosdem.org/2021/schedule/event/gnumes/ Janneke, that is probably most important step for GNU and free software in general. Once it becomes public enough and awareness is raised, many distributions will be nullified in terms of being fully free. I guess that Free System Distribution Guideliness will also need to be update when Guix gets its full bootstrap from source, so that every OS has to be bootstrapped from GNU Guix. Guix will become primary root distribution for every other distribution or otherwise they are not trusted. It does not mean that distributions are "secure" just because bootstrapping process exists, there is so much more work around that. It is very simple even to replicate the bootstrapping process and provide source from one malicious source, that is enough to corrupt the whole process for common users. However -- this does not exclude malicious codein various compilers, we still do not know if something is injected in a smart way. We still have to trust it. This is just one step forward to full inspection of the software. NSA, according to stories, has already ask Torvalds to inject backdoors. I don't think they will simply give up with their intentions. I have read it, and researching, but yet do not get how to start. I have downloaded stage0, is that the place to start? Jean
Re: Risks of deterministic builds (was: Re: Truth matters when writing software and selecting leaders)
* Martin [2021-04-06 12:22]: > > From practical viewpoint, among milions and millions of users, when it > > comes to validating compiler, they would have to validate the > > reproducible build with comparison to something. Benefits of > > reproducible builds thus depend of number of people validating it and > > reporting problems. It depends of publicity of problems and > > research. Small group of people may do the work, but they cannot > > possibly make sure to do the work for ALL distributions and for all > > people. Thus practically for an individual it means nothing, unless > > individual is highly skilled to verify internals of the compiler, and > > we have plethora of compilers on every single GNU/Linux operating > > system. Thus whole countries may be converted into spying backdoor > > teams by using marketing of reproducible builds of packages that > > people cannot really verified. Reproducible build of system is not > > yet reality. We hope for it in future. > Maybe freedom in "free software" shouldn't require from the code to be open > neither. Let's just blindly trust some saint developers who cannot even > control their own binaries. Actually today we are closer and closer to that > sad scenario like never before in the history, because in fact most of the > open-source and GNU "free software" nowadays base on blackboxed binary seeds > that cannot be verified by the users not even by the core developers. I say you are right there, only that irony is not really in place. I admire your perfectionism. - practically, majority of GNU/Linux and BSD-derivatives blindly trust their developers. It is how it is. Just few of them are actual developers who verify things and develop, and submit issues, find security problems and so on. We rely on our developers. - developers can to a degree control their binaries. It is questionable if they can boostrap compilers from pure sources, so they trust their upstream compiler providers like GNU GCC, or Haskell's origins, or other compilers. Guix is making effort and some other OS-es to make it boostrapable. - yes, with larger number of people using GNU/Linux we are closer and closer to scenario of blindly trusting our distributions. That is not good. Common users cannot anyway verify software. - You are right, that now, at this point of time, we should point out to that issue, as now it is important when it is not too late. Maybe it is too late for Haskell. I know for GCC is not too late as Guix can bootstrap it or almost bootstrap it. Not sure. If we don't point from today on about this issue, we will get serious problems in future. Awareness we need. Something practical has to be done about that. Did you contribute to Guix with your knowledge? -- Jean Take action in Free Software Foundation campaigns: https://www.fsf.org/campaigns Sign an open letter in support of Richard M. Stallman https://rms-support-letter.github.io/
Re: Risks of deterministic builds
Martin writes: > Maybe freedom in "free software" shouldn't require from the code to be > open neither. Let's just blindly trust some saint developers who > cannot even control their own binaries. Actually today we are closer > and closer to that sad scenario like never before in the history, > because in fact most of the open-source and GNU "free software" > nowadays base on blackboxed binary seeds that cannot be verified by > the users not even by the core developers. The bootstrappable project, GNU Mes and GNU Guix are working to fix that https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/ https://fosdem.org/2021/schedule/event/gnumes/ Greetings, Janneke -- Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
Re: Risks of deterministic builds (was: Re: Truth matters when writing software and selecting leaders)
On 4/6/21 7:40 AM, Jean Louis wrote: * Jacob Bachmeyer [2021-04-06 05:39]: Exploits are easier to develop when hardcoded offsets, virtual addresses, etc. can be used. In a "binary monoculture" environment, that is possible. This contributes to and worsens security problems in proprietary software, which is almost always distributed as a single identical set of binaries. If you have a source code that types of exploits are also easier to detect. Besides you can always compile your software with different flags then the one used by default. Reproducible-builds just gives you information that for a fixed environment you have the fixed binaries but usually the combinations of settings are very wide and it's only up to you how your binaries are distributed in the end. Reproducible builds are useful for validating the compiler, but there is a potential downside in that they make any exploit that can be found in the reproducibly built program much more reliable, since everyone will have exactly identical binaries. Note that this is an identical risk with binary distributions: if you simply install the binaries form Debian, an exploit can be tuned to Debian's version of that binary and it will work on your machine. So far debian is still one of the safest linux distribution in the world. Anyway even debian is giving you the option to compile all their software from source codes and again you can tune it as you like in your custom dev environment producing completely different binaries than others do. That is right. From practical viewpoint, among milions and millions of users, when it comes to validating compiler, they would have to validate the reproducible build with comparison to something. Benefits of reproducible builds thus depend of number of people validating it and reporting problems. It depends of publicity of problems and research. Small group of people may do the work, but they cannot possibly make sure to do the work for ALL distributions and for all people. Thus practically for an individual it means nothing, unless individual is highly skilled to verify internals of the compiler, and we have plethora of compilers on every single GNU/Linux operating system. Thus whole countries may be converted into spying backdoor teams by using marketing of reproducible builds of packages that people cannot really verified. Reproducible build of system is not yet reality. We hope for it in future. Maybe freedom in "free software" shouldn't require from the code to be open neither. Let's just blindly trust some saint developers who cannot even control their own binaries. Actually today we are closer and closer to that sad scenario like never before in the history, because in fact most of the open-source and GNU "free software" nowadays base on blackboxed binary seeds that cannot be verified by the users not even by the core developers.
Re: Risks of deterministic builds (was: Re: Truth matters when writing software and selecting leaders)
* Jacob Bachmeyer [2021-04-06 05:39]: > Exploits are easier to develop when hardcoded offsets, virtual addresses, > etc. can be used. In a "binary monoculture" environment, that is possible. > This contributes to and worsens security problems in proprietary software, > which is almost always distributed as a single identical set of binaries. > > Reproducible builds are useful for validating the compiler, but there is a > potential downside in that they make any exploit that can be found in the > reproducibly built program much more reliable, since everyone will have > exactly identical binaries. Note that this is an identical risk with binary > distributions: if you simply install the binaries form Debian, an exploit > can be tuned to Debian's version of that binary and it will work on your > machine. > > > -- Jacob That is right. >From practical viewpoint, among milions and millions of users, when it comes to validating compiler, they would have to validate the reproducible build with comparison to something. Benefits of reproducible builds thus depend of number of people validating it and reporting problems. It depends of publicity of problems and research. Small group of people may do the work, but they cannot possibly make sure to do the work for ALL distributions and for all people. Thus practically for an individual it means nothing, unless individual is highly skilled to verify internals of the compiler, and we have plethora of compilers on every single GNU/Linux operating system. Thus whole countries may be converted into spying backdoor teams by using marketing of reproducible builds of packages that people cannot really verified. Reproducible build of system is not yet reality. We hope for it in future. -- Jean Take action in Free Software Foundation campaigns: https://www.fsf.org/campaigns Sign an open letter in support of Richard M. Stallman https://rms-support-letter.github.io/
