Re: [opinion] CVE-patching is not sufficient for package security patching
On Thu, Mar 25, 2021 at 03:22:16PM +0100, Mathieu Othacehe wrote: > I recently added a new metric in Cuirass: "Builds count per machine > during the last day". Turns out the overdrive1 with its two workers > seems to outperform the hydra-guix-X running emulated builds on four > workers. That's good to know, thanks!
Re: [opinion] CVE-patching is not sufficient for package security patching
Hello, > Concretely, this would mean a Honeycomb LX2 or Ampere ALTRA workstation, > since I don't believe there are any other aarch64 workstations available > for sale. > > https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-workstation/ > https://store.avantek.co.uk/ampere-altra-64bit-arm-workstation.html I recently added a new metric in Cuirass: "Builds count per machine during the last day". Turns out the overdrive1 with its two workers seems to outperform the hydra-guix-X running emulated builds on four workers. As soon as the other overdrives are back online, the situation will hopefully be a tiny bit better. Buying and hosting other machines such as the ones you mentioned could also help here. The Wireguard tunnel between berlin an the overdrive1 works fine and configuring those machines with something similar to "hydra/modules/sysadmin/overdrive.scm" should be enough to add them as Cuirass workers. Thanks, Mathieu
Re: [opinion] CVE-patching is not sufficient for package security patching
On Wed, Mar 24, 2021 at 09:24:40PM +0100, Vincent Legoll wrote: > I already volunteered (privately) to host the same (1 or 2 WS power-class), > currently on ADSL uplink (so not for substitute distribution, only building), > FTTH in the future, no UPS though. The architecture of the build arm is such that substitutes are distributed centrally, so you wouldn't have to worry about slow uplink, as long as it's fast enough to pragmatically send build artifacts to the central node.
Re: [opinion] CVE-patching is not sufficient for package security patching
On Wed, 2021-03-24 at 21:24 +0100, Vincent Legoll wrote: > Hello, > > On Wed, Mar 24, 2021 at 8:51 PM Leo Famulari > wrote: > > > We bought a handful of Overdrive 1000 in the past (they are no > > > longer > > > sold), and hosting was always an obstacle. > > > > I volunteer to host one or two workstation-type 64-bit ARM > > machines. > > I already volunteered (privately) to host the same (1 or 2 WS power- > class), > currently on ADSL uplink (so not for substitute distribution, only > building), > FTTH in the future, no UPS though. > I also have space at home, ~3500Mbps down, 600Mbps up, no UPS (yet?). signature.asc Description: This is a digitally signed message part
Re: [opinion] CVE-patching is not sufficient for package security patching
Hello, On Wed, Mar 24, 2021 at 8:51 PM Leo Famulari wrote: > > We bought a handful of Overdrive 1000 in the past (they are no longer > > sold), and hosting was always an obstacle. > > I volunteer to host one or two workstation-type 64-bit ARM machines. I already volunteered (privately) to host the same (1 or 2 WS power-class), currently on ADSL uplink (so not for substitute distribution, only building), FTTH in the future, no UPS though. -- Vincent Legoll
Re: [opinion] CVE-patching is not sufficient for package security patching
On Tue, Mar 23, 2021 at 11:54:54PM +0100, Ricardo Wurmus wrote: > This seems to be a misunderstanding. The first step is to use the money > we already have but cannot exchange for hardware, because > > - finding appropriate hardware that you can actually buy is not easy > - hosting needs to be considered because we can’t just dump them in the > MDC data centre where most of the Intel servers are hosted. > > We bought a handful of Overdrive 1000 in the past (they are no longer > sold), and hosting was always an obstacle. > > While looking for ways to get the project some more money is certainly > worthwhile, it’s really not the pressing issue here. I volunteer to host one or two workstation-type 64-bit ARM machines. Concretely, this would mean a Honeycomb LX2 or Ampere ALTRA workstation, since I don't believe there are any other aarch64 workstations available for sale. https://www.solid-run.com/arm-servers-networking-platforms/honeycomb-workstation/ https://store.avantek.co.uk/ampere-altra-64bit-arm-workstation.html signature.asc Description: PGP signature
Re: [opinion] CVE-patching is not sufficient for package security patching
Joshua Branson writes: > raingloom writes: >> >> What about a Liberapay for Guix? Could also be used to pay developers. >> > > I'd be game for something like this. We could have a guix membership. > Drew Devault has a "secret irc" channel for paying patreons. Perhaps we > could advertise a guix membership on the guix site. Then, the Guix leadership would have to decide which Guix developers are worthy of funding, and how much. Those who are excluded may feel that their contributions are insufficiently valued, and therefore feel alienated. Sounds to me like it would open up a huge can of worms, so to speak. Regards, Mark
Re: [opinion] CVE-patching is not sufficient for package security patching
raingloom writes: > On Sat, 20 Mar 2021 12:19:11 +0100 > Ludovic Courtès wrote: > >> Hi, >> >> Mark H Weaver skribis: >> >> > Ultimately, I gave up. In my opinion, Guix has never achieved >> > usability as a desktop system on non-Intel systems. Therefore, the >> > Guix community is unable to attract many developers who want a >> > distro that supports non-Intel systems well. Our community has >> > thus become dominated by Intel users, and there's unsufficient >> > political will to adopt policies that would enable us to provide a >> > usable system for non-Intel users. >> >> A practical problem that’s been mentioned repeatedly is lack of >> ci.guix hardware for non-Intel architectures: please everyone, >> consider helping the project find either sponsors or companies that >> sell fitting hardware, along with a plan to host it and maintain it >> over time! >> >> Ludo’. >> > > What about a Liberapay for Guix? Could also be used to pay developers. This seems to be a misunderstanding. The first step is to use the money we already have but cannot exchange for hardware, because - finding appropriate hardware that you can actually buy is not easy - hosting needs to be considered because we can’t just dump them in the MDC data centre where most of the Intel servers are hosted. We bought a handful of Overdrive 1000 in the past (they are no longer sold), and hosting was always an obstacle. While looking for ways to get the project some more money is certainly worthwhile, it’s really not the pressing issue here. -- Ricardo
Re: [opinion] CVE-patching is not sufficient for package security patching
On Mon, Mar 22, 2021 at 02:44:04PM +0100, raingloom wrote: > What about a Liberapay for Guix? Could also be used to pay developers. Some of us already have Liberapay accounts.
Re: [opinion] CVE-patching is not sufficient for package security patching
raingloom writes: > > What about a Liberapay for Guix? Could also be used to pay developers. > I'd be game for something like this. We could have a guix membership. Drew Devault has a "secret irc" channel for paying patreons. Perhaps we could advertise a guix membership on the guix site. When someone signs up, it actually makes them an FSF member. I believe membership with the FSF includes perks such as: 5 email aliases, a federated XMPP account, and some other things... -- Joshua Branson (joshuaBPMan in #guix) Sent from Emacs and Gnus https://gnucode.me https://video.hardlimit.com/accounts/joshua_branson/video-channels https://propernaming.org "You can have whatever you want, as long as you help enough other people get what they want." - Zig Ziglar
Re: [opinion] CVE-patching is not sufficient for package security patching
On Sat, 20 Mar 2021 12:19:11 +0100 Ludovic Courtès wrote: > Hi, > > Mark H Weaver skribis: > > > Ultimately, I gave up. In my opinion, Guix has never achieved > > usability as a desktop system on non-Intel systems. Therefore, the > > Guix community is unable to attract many developers who want a > > distro that supports non-Intel systems well. Our community has > > thus become dominated by Intel users, and there's unsufficient > > political will to adopt policies that would enable us to provide a > > usable system for non-Intel users. > > A practical problem that’s been mentioned repeatedly is lack of > ci.guix hardware for non-Intel architectures: please everyone, > consider helping the project find either sponsors or companies that > sell fitting hardware, along with a plan to host it and maintain it > over time! > > Ludo’. > What about a Liberapay for Guix? Could also be used to pay developers.
Re: [opinion] CVE-patching is not sufficient for package security patching
Hi, Mark H Weaver skribis: > Ultimately, I gave up. In my opinion, Guix has never achieved usability > as a desktop system on non-Intel systems. Therefore, the Guix community > is unable to attract many developers who want a distro that supports > non-Intel systems well. Our community has thus become dominated by > Intel users, and there's unsufficient political will to adopt policies > that would enable us to provide a usable system for non-Intel users. A practical problem that’s been mentioned repeatedly is lack of ci.guix hardware for non-Intel architectures: please everyone, consider helping the project find either sponsors or companies that sell fitting hardware, along with a plan to host it and maintain it over time! Ludo’.
Re: [opinion] CVE-patching is not sufficient for package security patching
Sorry for duplicated email.. On Tue, 2021-03-16 at 19:19 -0400, Mark H Weaver wrote: > If not, it would be good to work toward the goal of making Guix > usable > on non-Intel systems. I'm sorry to say that, in my opinion, your > proposal would move us in the wrong direction to achieve that goal. We have been working with Chris Marusich, Efraim and numerous others on porting GNU Guix to PowerPC 64-bits, I think both are not contradictory. I have a Talos II desktop at home which once GNU Guix works on it will be the main machine I will use to contribute to GNU Guix (along with offloading to other machines for testing on other archs), so count on me to at the same time keep up on GNU Guix and test things on PowerPC 64-bits. I am of course concerned with any blob doing things I don't need (and introducing security risk) under the hood, that's why (along with strong software freedom imaginaries) I pre-ordered my RaptorCS Talos II machine in 2017 and that I have been trying since 2 years to bring PowerPC 64-bits to GNU Guix (also with numerous other folks who joined efforts most recently Chris Marusich who've been enormous help!). But I also want to be realistic that the major security risk in most computers today probably isnt the Intel ME or Intel AMT and that we also can do many other things in the system itself that reduces risk greatly. I'll be honest also, the IBM POWER chips have gotten much less security review than the Intel or AMD chips recently, so it's not because there's not as much security drama on IBM POWER that it doesnt have (maybe even more severe) issues :-) About the overall security of GNU Guix and the things we can do that don't involve keeping a fast-paced rythm to updating packages I see few things, right now GNU Guile is the center of all's GNU Guix security, I am not sure it received lots of security auditing, it's also written in part in a memory-unsafe language that is C, so there's probably some low hanging fruits there once some starts fuzzing it, I'm no big expert in fuzzing but I may try at some point. I think we can do many things complementary, prevention (sandboxing), mitigation (enabling hardening compiler flags, ..), AND code security patching. The first two don't require we keep a fast paced update rythm, also we may do the first two especially because we realize we can't do the latter at all time and I realize that, I just think we should always try to, at least, that's all. I am also a bit concerned with the idea that GNU Guix, GNU Shepherd etc. execute code from arbitrary files in many places, I am not sure all the security details have been reviewed here, it seems risky to me to have configuration files that allow executing arbitrary code, also GNU Guile seems to have a sandboxed evaluation mode, that's good. I like the freedom of arbitrary code evaluation anywhere gives us, but I also want to make sure it's actually secure to do so. Léo signature.asc Description: This is a digitally signed message part
Re: [opinion] CVE-patching is not sufficient for package security patching
On Tue, 2021-03-16 at 19:19 -0400, Mark H Weaver wrote: > That said, I strongly disagree that we should "never backport patches > ourselves in most cases". The only way to do that, while addressing > security flaws, would be to promptly update even our lowest-level > libraries in response to CVEs, of which there is a steady stream. Fortunately I think that lots of these core package upstreams also have good CVE-issuance practices. For the Glib care in particular, I think they are good, I consider acceptable to backport patches, everyone is doing it, upstream is cooperative and works towards that same goal. To everyone else in general, I understand we have to ship a working system, and I want that too, that's why I said we should "strive" to, but it doesnt mean we should break things, of course. By that I mean that we shouldnt leave packages unmaintained without updates for too long even without CVEs or other security notices issued. At some point, if a package is of no use, no users show up and it's painful to update, we should also consider removing the package or archiving it in a third party channel we could create like "guix-archive", "guix-ugly" or "guix-love-me-please". Léo signature.asc Description: This is a digitally signed message part
Re: [opinion] CVE-patching is not sufficient for package security patching
On Tue, Mar 16, 2021 at 07:19:59PM -0400, Mark H Weaver wrote: > Ultimately, I gave up. In my opinion, Guix has never achieved usability > as a desktop system on non-Intel systems. Therefore, the Guix community > is unable to attract many developers who want a distro that supports > non-Intel systems well. Our community has thus become dominated by > Intel users, and there's unsufficient political will to adopt policies > that would enable us to provide a usable system for non-Intel users. > > What do you think? Thanks, as always, for your well-reasoned message. Your story of your experience here, and what it means for Guix, computing security, and free software in general, is really valuable. I still think it's really unfortunate for the rest of us that you gave up, but I don't see how it could have gone differently. I agree with you that Guix moves too fast to support non-Intel architectures in its current state. My hope is that, within the next two years, there will be workstation aarch64 machines more widely available at comparable prices to Intel/AMD, and this will translate into more developer support for aarch64 in the years after that. Time will tell.
Re: [opinion] CVE-patching is not sufficient for package security patching
Hi Léo, Léo Le Bouter writes: > I would like to share some opinion I have on CVE-patching for non- > rolling release GNU/Linux distributions and why we should strive to > always update to the latest available releases or always follow > upstream supported release series and never backport patches ourselves > in most cases (some upstreams may have really good practices but these > are rare). > > A lot of security issues are patched silently in upstream projects > without ever getting a CVE, security issues may not be labeled as such > by upstreams for various reasons (fear of shame, belief to patch > something with no security impact while it has, bizarre security > through obscurity policy, ..). ... and I'll add that it can be a lot of work to evaluate, for a given bug, whether or not that bug is exploitable. Anyway, I agree that bugs fixed upstream are sometimes exploitable, even when they have not been explicitly identified as security flaws, and that this is a valid argument in favor of keeping our packages updated to the latest release. That said, I strongly disagree that we should "never backport patches ourselves in most cases". The only way to do that, while addressing security flaws, would be to promptly update even our lowest-level libraries in response to CVEs, of which there is a steady stream. Anyone with experience working on the 'staging' or 'core-updates' branches in Guix, or in the release process of Debian, will immediately recognize this proposal to be unrealistic. In practice, updating low-level or even mid-level libraries tends to cause breakage. This kind of integration breakage happens quite frequently, even on x86_64-linux, the architecture that most developers work on. It's *much* worse on other architectures. New upstream releases quite regularly cause breakage on less popular architectures. It is often left to distros such as Debian to fix these problems. Since you're interested in security, I'll now remind you that *all* modern Intel systems include another little computer inside them called the Management Engine, which is always on when the machine is plugged in (even when the computer is "off"), has it's own memory that the main CPU cannot see, runs a proprietary OS that the user cannot replace, has full access to the RAM and disk of the machine, and can talk to the network without the main CPU even seeing those packets. Are you comfortable with this? If not, it would be good to work toward the goal of making Guix usable on non-Intel systems. I'm sorry to say that, in my opinion, your proposal would move us in the wrong direction to achieve that goal. In my experience, Guix is already moving far too fast to be usable on less popular architectures. I have some knowledge of this. Years ago, I made a serious effort to make Guix usable on non-Intel systems. When Guix was young, I initiated its first two ports to non-Intel architectures: mips64el-linux and armhf-linux, and I tried to actually use Guix on those systems in practice. I found that my system was very frequently broken by upstream updates, and that we didn't have nearly enough developer energy to keep up with fixing those problems. I've come to believe that having Guix work well on non-Intel systems is, in practice, incompatible with the rate at which we update our packages. I'm not sure that even Debian would have enough energy to keep less popular architecures working well, given our practices. I raised this issue on guix-devel a few times over the years, but it became clear that the desire in this community to keep packages aggressively updated far outweighs any interest in supporting non-Intel systems. Ultimately, I gave up. In my opinion, Guix has never achieved usability as a desktop system on non-Intel systems. Therefore, the Guix community is unable to attract many developers who want a distro that supports non-Intel systems well. Our community has thus become dominated by Intel users, and there's unsufficient political will to adopt policies that would enable us to provide a usable system for non-Intel users. What do you think? Regards, Mark
Re: [opinion] CVE-patching is not sufficient for package security patching
On Tue, Mar 16, 2021 at 12:10:26PM +0100, Léo Le Bouter wrote: > For these reasons, I suggest that we always strive to update packages > to their latest versions and that I think it is security relevant to > always do so. Of course, new code could *introduce* new vulnerabilities > but I am not trying to debate this, it's that to the best of the > upstream's knowledge chances are that the latest version will contain > more security fixes than older versions (if that upstream is actually > maintaining the project). I agree that every new release can be considered to have fixed security problems. Please read the rest of my message while keeping in mind that I have spent *a lot* of time working on security in Guix over the years. We must keep in mind that there are other values besides security. Additionally, this kind of "security" mindset is a somewhat narrow way of considering the problem of secure computing. It's important to remember that security can be modeled with 3 factors: confidentiality, integrity, and availability. The 3rd factor is often overlooked. In terms of making a distro, there is a spectrum of approaches. At one end of the spectrum, there is something like PyPi, which is just a clearinghouse for upstream projects to distribute their code. Everything is always updated to the latest version. It does not provide a working system, even within the narrow world of "just Python"; there are broad incompatibilities among the latest versions of Python programs. On the other end is the approach of Red Hat and Debian. They laboriously filter the upstream software to provide stable operating systems. They do fix security bugs, but only after extensive validation that functionality is not changed. The result is useful but the cost is very high. Guix has always been in the middle, along with other rolling release distros, and I think that's a good place for us to be. With our superior tooling we can be "more stable than rolling release" while also "moving faster than stable". It's instructive to consider the Linux kernel. They release about once a week, and every release fixes serious but unpublicized bugs. At the time they are announcing the release, they are already aware of other serious bugs, that might be fixed in the next release. It sounds terrible, and yet Linux is by far the most popular and useful general-purpose operating system. The world of computing, which is based on Linux, continues to serve our civilization well. That's because the most important thing value in Linux development is to not break anything for users; security is not the top priority, but just another important thing to consider. I think that, as an operating system distro, we must adopt a similar mindset, and be careful not to sacrifice too much for an abstract sense of security based on fixing CVEs, which are an arbitrary system that have little bearing on utility or safety in the real world, which is where security matters. Of course we should fix CVEs, but we must also recognize that rushing too much reduces stability and availability. We have to weigh the costs and benefits every time. signature.asc Description: PGP signature
Re: [opinion] CVE-patching is not sufficient for package security patching
On Tue, 2021-03-16 at 12:17 +0100, Jonathan Brielmaier wrote: > I think the only two reasons against that are: time and > CI/rebuilding. I > think thats the reason why stuff like Gnome and others lower in the > dependency tree are lacking behind... Being non-FHS and non-systemd > makes updates for those stuff not easier and is maybe the third > reason/root issue... I agree with all 3 points. I have hope however that we can develop better tooling over time to ease the burden on us so we can devote more time to tasks that actually absolutely **require** our human oversight to be done. And then even without an increase in the contributor base we can avoid lagging behind on these updates. signature.asc Description: This is a digitally signed message part
Re: [opinion] CVE-patching is not sufficient for package security patching
On 16.03.21 12:10, Léo Le Bouter wrote: For these reasons, I suggest that we always strive to update packages to their latest versions and that I think it is security relevant to always do so. Of course, new code could *introduce* new vulnerabilities but I am not trying to debate this, it's that to the best of the upstream's knowledge chances are that the latest version will contain more security fixes than older versions (if that upstream is actually maintaining the project). I think the only two reasons against that are: time and CI/rebuilding. I think thats the reason why stuff like Gnome and others lower in the dependency tree are lacking behind... Being non-FHS and non-systemd makes updates for those stuff not easier and is maybe the third reason/root issue...
[opinion] CVE-patching is not sufficient for package security patching
Hello! I would like to share some opinion I have on CVE-patching for non- rolling release GNU/Linux distributions and why we should strive to always update to the latest available releases or always follow upstream supported release series and never backport patches ourselves in most cases (some upstreams may have really good practices but these are rare). A lot of security issues are patched silently in upstream projects without ever getting a CVE, security issues may not be labeled as such by upstreams for various reasons (fear of shame, belief to patch something with no security impact while it has, bizarre security through obscurity policy, ..). For these reasons, I suggest that we always strive to update packages to their latest versions and that I think it is security relevant to always do so. Of course, new code could *introduce* new vulnerabilities but I am not trying to debate this, it's that to the best of the upstream's knowledge chances are that the latest version will contain more security fixes than older versions (if that upstream is actually maintaining the project). In many cases, browsing through the commit history of some popular projects can uncover security issues not publicized through any security mailing lists or CVEs anywhere, this is unfortunately quite common. We cannot possibly monitor the commit history (and code) of every single project to backport fixes when we would need to. It is better for us to always strive to use the latest versions even when it requires us to do more far-reaching changes because of dependents/dependencies. Let me know what you think! Léo signature.asc Description: This is a digitally signed message part
