Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Wed, Jan 24, 2024 at 09:37:27AM +0100, Simon Josefsson wrote: > Simon Josefsson writes: > > >> > My naive approach on how to fix a security problem in package X > >> > which is > >> > statically embedded into other packages A, B, C, ... would be to > >> > rebuild > >> > the transitive closure of all packages that Build-Depends on X and > >> > publish a security update for all those packages. > ... > > I realized that there is one problem with my approach: consider if > > package A was built via Build-Depends package B of version X and that > > later package B is upgraded to X+1 in Debian. Then if a security > > problem happens in B we need to rebuild A. It may be that package A no > > longer builds due to some incompatibility between version X and X+1 of > > B. This would not be noticed until a full rebuild of an archive is > > done, or when the security teams wants to rebuild the transitive > > closure of the Build-Depends graph for a package. > > Having reflected a bit, and learned through my own experience and > others' insights [1] that Go Build-Depends are not transitive, I'd like > to update my proposal on how to handle a security bug in any Go/Rust/etc > package and the resulting package rebuilds: > > To fix a security problem in package X, which is used during build > (through statical linking, vendoring, or some other mechanism) to > build package A, B, C, ... you need to rebuild all packages that > Build-Depends on X (lets call this step 1) and all packages that > Build-Depends on any of the packages that will be rebuilt during step > 1. This rebuild process is iterated until no more packages remains to > be rebuild, and all packages have been rebuilt using newly rebuild > packages. If there are cyclical dependencies (which unfortunately are > common), you have to loop until all packages have been rebuilt with a > clean dependency chain, and detect the loop and stop rebuilding. If > there are FTBFS errors during the rebuilds, this will have to be > patched too. > > Yes, for a low-level Go package (e.g., golang-golang-x-net-dev), this > will mean rebuilding almost all of the Go packages in Debian and publish > them in a security advisory. > > This algorithm can be optimized (i.e., reduce the number of packages to > publish in an advisory) by either of: > > 1) using information from Built-Using: (which was not designed for >this purpose, so this is fragile) or *.buildinfo. > > 2) by dropping all 'Architecture: all' packages that does not embedd >the buggy code. > > The last optimization 2) would reduce the number of Go packages to > publish significantly, as it would drop most golang-*-dev packages. I > think this actually makes this process feasible in practice, as there > are relatively few binary packages written in Go. > > This method applies to non-Go/Rust too, if there are such security > problems and reverse build chains. > > I think all of this (except maybe the optimization 2) which requires > code comparisons) can be automated in a GitLab pipeline for higher > confidence of the result. Here's some rough thoughts and numbers for the (packaged) Rust ecosystem.. Currently in unstable we have ~100 packages shipping binaries written in Rust that are packaged using the "stock" packaging schema (that's debcargo/debcargo-conf + dh-cargo + our cargo wrapper) or Jonas' forked one (which use a slightly different, but mostly compatible approach), which means dh-cargo-built-using is called and encodes the static linking information in the binary package in X-Cargo-Built-Using[0]. Let's add another 50[1] that are using Rust in some fashion and thus (potentially) contain statically linked Rust code from other source packages, which are for whatever reason currently not emitting this information - some examples: src:cargo, src:rustc - these both vendor their deps and need special handling anyhow, firefox and soon chromium (similar AFAIK), but also more relevant, python3-cryptography and friends, or parts of Gnome that are newly or re-written in Rust (these could and should emit the information, but currently don't). These are the only packages (besides the librust-foo-dev package triggering a rebuild) that need a rebuild right now if a security fix needs to be deployed. But thankfully we don't need to rebuild 1xx Rust-using packages whenever any crate has a security fix, we only need to do so if they transitively build-depend on the vulnerable (now fixed) package/crate. This in effect also means that (for simple cases where the vulnerability is not spread across multiple crates) we can rebuild the vulnerable (lib) crate first, and then schedule independent rebuilds of all binary-shipping packages that statically link it. I attached a list doing a rough mapping of source package to number of such packages needing a rebuild if the source package in question gets a security fix. In my experience (both packaging Rust things in Debian, bu
Re: Limited security support for Go/Rust? Re ssh3
On Wed, Jan 24, 2024 at 11:40:20PM +, Wookey wrote: > If it only done for security issues, rather than routinely, then that > shouldn't be that much extra load (does anyone have any idea how much > extra building we are talking about? Is it trivial, or huge?) If we do those rebuilds in stable security we have several bottlenecks: - We need to get all the other packages into security, need some way to trigger rebuilds (I still think we need to kill the current way of BinNMU alltogether) - Those can only build after the fixed package was released (or at least completely built) - DSA are supposed to be available for all updates, so we now need to list hundred of packages. - The separate security archive is severely space limited. - SRM like to look at all updates as well. Bastian -- Worlds are conquered, galaxies destroyed -- but a woman is always a woman. -- Kirk, "The Conscience of the King", stardate 2818.9
Re: Limited security support for Go/Rust? Re ssh3
On 2024-01-16 10:59 +0100, Simon Josefsson wrote: > My naive approach on how to fix a security problem in package X which is > statically embedded into other packages A, B, C, ... would be to rebuild > the transitive closure of all packages that Build-Depends on X and > publish a security update for all those packages. > > What is the problem with that approach to handle security problems in a > Go package for trixie? Well, the reason we have not done this in the past is that the rebuild propogation needed potentially adds significant load to the buildds, and we used to have barely enough capacity to keep up with uploads for some architectures. I think that situation is quite a lot better these days, at least for release architectures, but I'm not sure how much slack there is in the system across the board? Other distros that routinely rebuild everything don't support anything like as many architectures. If it only done for security issues, rather than routinely, then that shouldn't be that much extra load (does anyone have any idea how much extra building we are talking about? Is it trivial, or huge?) Wookey -- Principal hats: Debian, Wookware, ARM http://wookware.org/ signature.asc Description: PGP signature
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Jan 24, Peter Pentchev wrote:
> This might be a minority, optimistic, rose-tinted-glasses kind of
> opinion, but I believe that the state of the Rust ecosystem today
> (I have no experience with the Go one) is quite similar to what Perl and
> Python modules were 25, 20, bah, even 15 years ago. Gradually, with time,
I am not familiar with the Python ecosystem, but I have been writing
Perl and packaging software with Perl dependencies for over 25 years and
I can confidently say that this is not true.
Perl libraries ("modules") generally never had the API instability that
I have seen in Rust libraries (but much less in Go, I believe).
In my experience forward compatibility has always been very important in
the Perl ecosystem.
BTW, for the past couple of years I have been presenting to my other
community, the network operators, about some of my Debian work and these
problems with integrating complex Rust software in distributions, e.g.
https://www.linux.it/~md/text/rpki-validators-euroix2023.pdf .
--
ciao,
Marco
signature.asc
Description: PGP signature
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Wed, Jan 24, 2024 at 01:01:34PM +, Luca Boccassi wrote: > On Wed, 24 Jan 2024 at 12:26, Johannes Schauer Marin Rodrigues > wrote: > > > > Hi, > > > > Quoting Luca Boccassi (2024-01-24 12:59:38) > > > There's always option B: recognize that the Rust/Go ecosystems are not > > > designed to be compatible with the Linux distributions model, and are > > > instead > > > designed to be as convenient as possible for a _single_ application > > > developer > > > and its users - at the detriment of everybody else - and for large > > > corporations that ship a handful of applications with swathes of engineers > > > that can manage the churn, and it is not made nor intended to scale to > > > ~60k > > > packages or whatever number we have today in unstable. And simply stop > > > attempting to merge these two incompatible ecosystems against their will, > > > at > > > the very least until and unless they reach feature and stability parity > > > with > > > the C/C++ ecosystems - stable API/ABI and dynamic linking by default. > > > > > > There are many ways to ship applications today that are much better > > > suited for these models, like Flatpak, where the maintenance burden is > > > shifted onto those who choose to opt in to such ecosystems. > > > > how does that work for those applications that require rust, go and friends? > > Are you proposing that everything that needs them should be be distributed > > by a > > flatpak or similar mechanism instead? > > Those applications are not shipped in the distribution. If somebody > wants to use them, they'll have to figure it out, just like for > everything else that is not shipped in the distribution, which is > already a subset of all available software in the world. > > > Just a few days ago I tried building mesa from experimental (otherwise there > > are severe graphics glitches on my platform) on bookworm and everything > > worked > > except its rust build dependencies. I had no luck trying to backport those > > parts of rust that I needed and even if it had worked, it would've meant > > backporting dozens of rust packages just to have a backport of mesa. It > > seemed > > way simpler to just disable the mesa component that requires rust and call > > it a > > day. But that probably doesn't work for all applications and maybe sometimes > > the component written in rust is actually what you want. And in case of > > mesa, a > > flatpak of it probably would also not've helped as that would've meant that > > I > > need to run everything that I want to run with a more modern mesa based on > > that > > flatpak, right? > > > > So how is option B a solution in practice? > > You have found first-hand the exact problem with this ecosystem. Now > try to imagine doing that for 30k packages, all vendoring, pinning and > static linking against each other at different, incompatible versions. > Again, the solution in practice is to simply disable or patch out > those components, and if somebody needs them, they can complain to > upstream and ask them for a solution, if there is one. If there isn't, > though luck. And yes, that is not great - but neither are all other > options, so it seems much wiser to me to push responsibility where it > belongs - with the upstreams choosing to use such ecosystems, and the > owner of said ecosystems choosing to make them incompatible with the > distribution model, as they are in the best position to solve the > problem(s) that happen due to their design choices. This might be a minority, optimistic, rose-tinted-glasses kind of opinion, but I believe that the state of the Rust ecosystem today (I have no experience with the Go one) is quite similar to what Perl and Python modules were 25, 20, bah, even 15 years ago. Gradually, with time, the module authors realized that if they wanted people to use their modules, they would eventually have to settle on a stable API and only make incompatible changes very rarely. With time, with this realization trickling up and down across the most used libraries, things slowly start to get better. Yes, even now there are what can only be called "transitions" in the Perl and Python Debian packaging - some often-used module makes an incompatible change and the packagers need to wait until all the other packaged modules have either caught up or it has somehow been proven through testing that some of the dependent modules are not really affected by the incompatible change. In my experience, nowadays this happens much, much more rarely than 10 or 15 years ago. I have no idea how long it will take the Rust ecosystem to realize that. I know that some of the most widely used modules have already done that, some of them have been at the same 0.x.* or even 1.* version for years (yes, really, years). So... here's hoping. G'luck, Peter -- Peter Pentchev r...@ringlet.net r...@debian.org p...@storpool.com PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C F115
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On 2024-01-24 13:26:49 +0100 (+0100), Johannes Schauer Marin Rodrigues wrote: [...] > how does that work for those applications that require rust, go > and friends? Are you proposing that everything that needs them > should be be distributed by a flatpak or similar mechanism > instead? > > Just a few days ago I tried building mesa from experimental > (otherwise there are severe graphics glitches on my platform) on > bookworm and everything worked except its rust build dependencies. > I had no luck trying to backport those parts of rust that I needed > and even if it had worked, it would've meant backporting dozens of > rust packages just to have a backport of mesa. [...] Another practical example is Python applications/libs increasingly integrating performance or security critical routines (re)written in Rust. The python-cryptography source package in sid build-depends on cargo and a dozen Rust libs. I didn't bother to pull a rdeps list for it, but can guarantee a vast swath of the python3-* packages in Debian depend on python3-cryptography, and this particular problem will only get worse as the CPython core devs see the rising popularity of the externally-developed cryptography library as a good reason to strip any remnants of cryptographic modules and bindings from the stdlib. -- Jeremy Stanley signature.asc Description: PGP signature
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Wed, 24 Jan 2024 at 12:26, Johannes Schauer Marin Rodrigues wrote: > > Hi, > > Quoting Luca Boccassi (2024-01-24 12:59:38) > > There's always option B: recognize that the Rust/Go ecosystems are not > > designed to be compatible with the Linux distributions model, and are > > instead > > designed to be as convenient as possible for a _single_ application > > developer > > and its users - at the detriment of everybody else - and for large > > corporations that ship a handful of applications with swathes of engineers > > that can manage the churn, and it is not made nor intended to scale to ~60k > > packages or whatever number we have today in unstable. And simply stop > > attempting to merge these two incompatible ecosystems against their will, at > > the very least until and unless they reach feature and stability parity with > > the C/C++ ecosystems - stable API/ABI and dynamic linking by default. > > > > There are many ways to ship applications today that are much better > > suited for these models, like Flatpak, where the maintenance burden is > > shifted onto those who choose to opt in to such ecosystems. > > how does that work for those applications that require rust, go and friends? > Are you proposing that everything that needs them should be be distributed by > a > flatpak or similar mechanism instead? Those applications are not shipped in the distribution. If somebody wants to use them, they'll have to figure it out, just like for everything else that is not shipped in the distribution, which is already a subset of all available software in the world. > Just a few days ago I tried building mesa from experimental (otherwise there > are severe graphics glitches on my platform) on bookworm and everything worked > except its rust build dependencies. I had no luck trying to backport those > parts of rust that I needed and even if it had worked, it would've meant > backporting dozens of rust packages just to have a backport of mesa. It seemed > way simpler to just disable the mesa component that requires rust and call it > a > day. But that probably doesn't work for all applications and maybe sometimes > the component written in rust is actually what you want. And in case of mesa, > a > flatpak of it probably would also not've helped as that would've meant that I > need to run everything that I want to run with a more modern mesa based on > that > flatpak, right? > > So how is option B a solution in practice? You have found first-hand the exact problem with this ecosystem. Now try to imagine doing that for 30k packages, all vendoring, pinning and static linking against each other at different, incompatible versions. Again, the solution in practice is to simply disable or patch out those components, and if somebody needs them, they can complain to upstream and ask them for a solution, if there is one. If there isn't, though luck. And yes, that is not great - but neither are all other options, so it seems much wiser to me to push responsibility where it belongs - with the upstreams choosing to use such ecosystems, and the owner of said ecosystems choosing to make them incompatible with the distribution model, as they are in the best position to solve the problem(s) that happen due to their design choices.
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
Hi, Quoting Luca Boccassi (2024-01-24 12:59:38) > There's always option B: recognize that the Rust/Go ecosystems are not > designed to be compatible with the Linux distributions model, and are instead > designed to be as convenient as possible for a _single_ application developer > and its users - at the detriment of everybody else - and for large > corporations that ship a handful of applications with swathes of engineers > that can manage the churn, and it is not made nor intended to scale to ~60k > packages or whatever number we have today in unstable. And simply stop > attempting to merge these two incompatible ecosystems against their will, at > the very least until and unless they reach feature and stability parity with > the C/C++ ecosystems - stable API/ABI and dynamic linking by default. > > There are many ways to ship applications today that are much better > suited for these models, like Flatpak, where the maintenance burden is > shifted onto those who choose to opt in to such ecosystems. how does that work for those applications that require rust, go and friends? Are you proposing that everything that needs them should be be distributed by a flatpak or similar mechanism instead? Just a few days ago I tried building mesa from experimental (otherwise there are severe graphics glitches on my platform) on bookworm and everything worked except its rust build dependencies. I had no luck trying to backport those parts of rust that I needed and even if it had worked, it would've meant backporting dozens of rust packages just to have a backport of mesa. It seemed way simpler to just disable the mesa component that requires rust and call it a day. But that probably doesn't work for all applications and maybe sometimes the component written in rust is actually what you want. And in case of mesa, a flatpak of it probably would also not've helped as that would've meant that I need to run everything that I want to run with a more modern mesa based on that flatpak, right? So how is option B a solution in practice? Thanks! cheers, josch signature.asc Description: signature
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Wed, 24 Jan 2024 at 11:42, Simon Josefsson wrote: > > Simon Josefsson writes: > > >> > My naive approach on how to fix a security problem in package X > >> > which is > >> > statically embedded into other packages A, B, C, ... would be to > >> > rebuild > >> > the transitive closure of all packages that Build-Depends on X and > >> > publish a security update for all those packages. > ... > > I realized that there is one problem with my approach: consider if > > package A was built via Build-Depends package B of version X and that > > later package B is upgraded to X+1 in Debian. Then if a security > > problem happens in B we need to rebuild A. It may be that package A no > > longer builds due to some incompatibility between version X and X+1 of > > B. This would not be noticed until a full rebuild of an archive is > > done, or when the security teams wants to rebuild the transitive > > closure of the Build-Depends graph for a package. > > Having reflected a bit, and learned through my own experience and > others' insights [1] that Go Build-Depends are not transitive, I'd like > to update my proposal on how to handle a security bug in any Go/Rust/etc > package and the resulting package rebuilds: There's always option B: recognize that the Rust/Go ecosystems are not designed to be compatible with the Linux distributions model, and are instead designed to be as convenient as possible for a _single_ application developer and its users - at the detriment of everybody else - and for large corporations that ship a handful of applications with swathes of engineers that can manage the churn, and it is not made nor intended to scale to ~60k packages or whatever number we have today in unstable. And simply stop attempting to merge these two incompatible ecosystems against their will, at the very least until and unless they reach feature and stability parity with the C/C++ ecosystems - stable API/ABI and dynamic linking by default. There are many ways to ship applications today that are much better suited for these models, like Flatpak, where the maintenance burden is shifted onto those who choose to opt in to such ecosystems.
Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
Simon Josefsson writes: >> > My naive approach on how to fix a security problem in package X >> > which is >> > statically embedded into other packages A, B, C, ... would be to >> > rebuild >> > the transitive closure of all packages that Build-Depends on X and >> > publish a security update for all those packages. ... > I realized that there is one problem with my approach: consider if > package A was built via Build-Depends package B of version X and that > later package B is upgraded to X+1 in Debian. Then if a security > problem happens in B we need to rebuild A. It may be that package A no > longer builds due to some incompatibility between version X and X+1 of > B. This would not be noticed until a full rebuild of an archive is > done, or when the security teams wants to rebuild the transitive > closure of the Build-Depends graph for a package. Having reflected a bit, and learned through my own experience and others' insights [1] that Go Build-Depends are not transitive, I'd like to update my proposal on how to handle a security bug in any Go/Rust/etc package and the resulting package rebuilds: To fix a security problem in package X, which is used during build (through statical linking, vendoring, or some other mechanism) to build package A, B, C, ... you need to rebuild all packages that Build-Depends on X (lets call this step 1) and all packages that Build-Depends on any of the packages that will be rebuilt during step 1. This rebuild process is iterated until no more packages remains to be rebuild, and all packages have been rebuilt using newly rebuild packages. If there are cyclical dependencies (which unfortunately are common), you have to loop until all packages have been rebuilt with a clean dependency chain, and detect the loop and stop rebuilding. If there are FTBFS errors during the rebuilds, this will have to be patched too. Yes, for a low-level Go package (e.g., golang-golang-x-net-dev), this will mean rebuilding almost all of the Go packages in Debian and publish them in a security advisory. This algorithm can be optimized (i.e., reduce the number of packages to publish in an advisory) by either of: 1) using information from Built-Using: (which was not designed for this purpose, so this is fragile) or *.buildinfo. 2) by dropping all 'Architecture: all' packages that does not embedd the buggy code. The last optimization 2) would reduce the number of Go packages to publish significantly, as it would drop most golang-*-dev packages. I think this actually makes this process feasible in practice, as there are relatively few binary packages written in Go. This method applies to non-Go/Rust too, if there are such security problems and reverse build chains. I think all of this (except maybe the optimization 2) which requires code comparisons) can be automated in a GitLab pipeline for higher confidence of the result. /Simon [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806481#58 signature.asc Description: PGP signature
Re: Limited security support for Go/Rust? Re ssh3
Adam Borowski writes: > On Mon, Jan 15, 2024 at 10:17:17AM +0100, Bastian Blank wrote: >> On Sun, Jan 14, 2024 at 04:24:57PM +0100, Simon Josefsson wrote: >> > Isn't that what the text refers to? Vendoring and static linking are >> > two examples of the same problem that the security team may encounter. >> >> We accept vendoring of autoconf/automake/gnulib distro wide. > > We _did_ accept that in the past, but these days you get smacked with a RC > bug for not building from source. What do you mean? Can you show me _any_ package in Debian that re-bootstrap itself using gnulib during package build? If by source you mean the source code in gnulib, not the vendored version shipping with many packages. If a security bug is found in gnulib code, one approach to fix it would be to patch the Debian gnulib package, and then automatically rebuild all packages that uses that gnulib function. I believe it is rare for packages in Debian to re-bootstrap itself from gnulib sources. Look at coreutils, sed, grep, tar, wget, libidn2, etc, none of them do that. Certainly not a RC bug. The situation now when a security bug in gnulib is found, you will have to patch all packages using that code manually per-package. >> The vendoring of gnulib, well, is old and maybe you could >> show that it is a problem in the sources that have it, aka they don't >> handle security fixes and at the same time don't change the library. > > Gnulib has not been useful for ages, thus packages still shipping vendored > copies are harmless -- functions that gnulib was meant to provide > implementation for were missing on ancient unices like HP-UX or SCO that > are long dead by now. A glance at recent commits in gnulib shows a lot of > retrocomputing names: Windows, OS/2, MacOS 10.5, AIX, on hardware of that > level of recency. It's not used for new ports: the most recent reference > to riscv in commit messages is from _2018_. I think you underestimate how often gnulib is used, and how well it works on modern platforms, and how much gnulib code is used that's not in glibc or other system shared libraries. I assume coreutils, sed, tar, gzip etc were important bootstrapping packages for riscv, and they all build fine thanks in large extent due to gnulib being written in a architecture independent way. /Simon signature.asc Description: PGP signature
Re: Limited security support for Go/Rust? Re ssh3
On Mon, Jan 15, 2024 at 10:17:17AM +0100, Bastian Blank wrote: > On Sun, Jan 14, 2024 at 04:24:57PM +0100, Simon Josefsson wrote: > > Isn't that what the text refers to? Vendoring and static linking are > > two examples of the same problem that the security team may encounter. > > We accept vendoring of autoconf/automake/gnulib distro wide. We _did_ accept that in the past, but these days you get smacked with a RC bug for not building from source. > Please show practical problems with it? * not working on new archs (most of the work of making sure autoconfage gets rebuilt was done by arm64 porters when it was a new thing) * not being able to fix bugs in autoconfage * failing to adapt to changes in the toolchain > The vendoring of gnulib, well, is old and maybe you could > show that it is a problem in the sources that have it, aka they don't > handle security fixes and at the same time don't change the library. Gnulib has not been useful for ages, thus packages still shipping vendored copies are harmless -- functions that gnulib was meant to provide implementation for were missing on ancient unices like HP-UX or SCO that are long dead by now. A glance at recent commits in gnulib shows a lot of retrocomputing names: Windows, OS/2, MacOS 10.5, AIX, on hardware of that level of recency. It's not used for new ports: the most recent reference to riscv in commit messages is from _2018_. Thus, I say the problem with vendored libraries is mostly unrelated to that of static linking, and thus they don't need to share a solution. > Here the problem is embedded into the language oekosystem itself. It is > not a choice of the software author to do static linking. It's mostly an issue with the deployment scheme at Google: I heard they recompile all their software and reinstall it on their fleet EVERY WEEK. This is about the only case where non-special-case (/sbin/ldconfig etc) static linking works. Then they shared their internal project with the world (good) without caring how it affects others (bad). Unlike them, we can't rebuild the world every time a bug is fixed (security or not). Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁'Russkiy voyennyi korabl, idi nakhuy' ⢿⡄⠘⠷⠚⠋⠀ ⠈⠳⣄
Re: Limited security support for Go/Rust? Re ssh3
El 16/01/24 a las 17:43, Simon Josefsson escribió: > Bastian Blank writes: > > > On Tue, Jan 16, 2024 at 10:59:30AM +0100, Simon Josefsson wrote: > >> Rebuilding a bit more than what is strictly needed sounds fine as a > >> first solution to me. > > > > Building maybe. But how do you want to publish them? The security > > archive is not made to handle that. > > What is the limitation? Is it human time involved to rebuild and QA > packages? Or the technical infrastructure related to publish security > fixes? Or the hosting infrastructure to deliver the package upgrades > via security.debian.org? Or something else? > I cannot talk for the security team, but there are infrastructure limitations (as stated in the release notes): https://lists.debian.org/debian-lts/2023/12/msg00035.html signature.asc Description: PGP signature
Re: Limited security support for Go/Rust? Re ssh3
> "Simon" == Simon Josefsson writes: Simon> Right, these are slightly different technical problems, but Simon> as far as the brief discussion in the release notes -- Simon> https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html#golang-static-linking Simon> -- I think the relevant aspect is identical: package X in Simon> Debian contains an embedded copy of code whose corresponding Simon> source code is in package Y in Debian, and fixing a security Simon> problem in Y will require rebuilding X and the entire Simon> dependency chain between X and Y that carry the code that Simon> ends up in X. That's what happens with static linking. I think the vendoring situation is worse because rebuilding x and the dependencies from x to y doesn't automatically pick up the fix. So, while I agree with Bastian that you described vendoring and not static linking, I think the key aspects of security support are *better* for static linking than vendoring. So, I'm not really sure why it's reasonable for you to be required to come up with static linking examples from C. Although examples of header-only c++ libraries or other libraries that use significant inlining are easy to find.
Re: Limited security support for Go/Rust? Re ssh3
Bastian Blank writes: > On Tue, Jan 16, 2024 at 10:59:30AM +0100, Simon Josefsson wrote: >> Rebuilding a bit more than what is strictly needed sounds fine as a >> first solution to me. > > Building maybe. But how do you want to publish them? The security > archive is not made to handle that. What is the limitation? Is it human time involved to rebuild and QA packages? Or the technical infrastructure related to publish security fixes? Or the hosting infrastructure to deliver the package upgrades via security.debian.org? Or something else? >> My naive approach on how to fix a security problem in package X which is >> statically embedded into other packages A, B, C, ... would be to rebuild >> the transitive closure of all packages that Build-Depends on X and >> publish a security update for all those packages. > > So if a fix to the net/tls module of go shows up (happens from time to > time), all go packages needs to be rebuilt? All Go packages that are affected by the security problem of net/tls, yes. That seems to be how the Go eco-system want things to be, the static linking nature is by design. Whether that is a good or bad design can be discussed, but it seems intentional and won't go away. My main point is that we have this situation in Debian already. Maybe vendoring via gnulib is a better example than config.h or statical linking though. Having being more exposed to the packaging aspects of Go projects in Debian recently, I am more sympathetic towards being conservative about offering security claims for these packages though. I wouldn't envy anyone trying to resolve a security vulnerability for a Go package in bookworm with many reverse dependencies. OTOH, I don't think it is sustainable to ignore security vulnerabilities in software that we ship. We just have to find a technical solution to solve whatever problems prevent us from being able to do that. /Simon signature.asc Description: PGP signature
Re: Limited security support for Go/Rust? Re ssh3
On 1/16/24 17:20, Simon Josefsson wrote: it seems that many people think that "Built-Using" can be used to express static linking (including yours truly, even though i *know* that it is meant for license compliance only). which makes me wonder: probably we should have an additional field that expresses such static linking (and therefore would trigger a rebuild when the dependency changes). or we could finally accept that many¹ people would just use "Built-Using" for this anyhow, and explicitly allow such use. Would that be better or worse than making *.buildinfo files more generally available and required? Buildinfo files appears to have some traction already, and it seems like they could help address the same problem. my understanding¹ of buildinfo files is, that they collect the exact information against which packages a given binary package was built. but that doesn't really help us with static linking (or license compliance as in Built-Using), as the binary might just use whatever parts of it's build-dependencies, and do e.g. dynamic linking or just run a given tool (e.g. 'ninja'). we can already track the required dependencies with Build-Depends, so we could also just rebuild all reverse dependencies. but afaiu, we want a more specific approach to be able to rebuild only those packages that would actually benefit from that (as opposed to a full archive rebuild because 'make' was uploaded for a policy version bump). gfmdasr IOhannes ¹ which is pretty dim on this topic; so do not hesitate to enlighten me OpenPGP_0xB65019C47F7A36F8.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Limited security support for Go/Rust? Re ssh3
"IOhannes m zmölnig (Debian GNU|Linux)" writes: > On 1/16/24 13:56, Jérémy Lal wrote: >>> >>> As Built-Using is for license compliance only, no? >>> >>> See >>> >>> https://www.debian.org/doc/debian-policy/ch-relationships.html#additional-source-packages-used-to-build-the-binary-built-using >> Indeed, thanks for the link. >> > > it seems that many people think that "Built-Using" can be used to > express static linking (including yours truly, even though i *know* > that it is meant for license compliance only). > > which makes me wonder: probably we should have an additional field > that expresses such static linking (and therefore would trigger a > rebuild when the dependency changes). > or we could finally accept that many¹ people would just use > "Built-Using" for this anyhow, and explicitly allow such use. Would that be better or worse than making *.buildinfo files more generally available and required? Buildinfo files appears to have some traction already, and it seems like they could help address the same problem. Unfortunately *.buildinfo still seems hard to access reliably and their integrity aren't protected by the archive-wide InRelease signature, if I understand correctly. /Simon signature.asc Description: PGP signature
Re: Limited security support for Go/Rust? Re ssh3
On Tue, Jan 16, 2024 at 04:44:14PM +0100, IOhannes m zmölnig (Debian GNU|Linux) wrote: > On 1/16/24 13:56, Jérémy Lal wrote: > > > > > > As Built-Using is for license compliance only, no? > > > > > > See > > > > > > https://www.debian.org/doc/debian-policy/ch-relationships.html#additional-source-packages-used-to-build-the-binary-built-using > > > > Indeed, thanks for the link. > > > > it seems that many people think that "Built-Using" can be used to express > static linking (including yours truly, even though i *know* that it is meant > for license compliance only). > > which makes me wonder: probably we should have an additional field that > expresses such static linking (and therefore would trigger a rebuild when > the dependency changes). There's been some effort to move to a Static-Built-Using field to express that semantic, and leave Built-Using as its intended purpose related to licensing. Cheers, -- James GPG Key: 4096R/91BF BF4D 6956 BD5D F7B7 2D23 DFE6 91AE 331B A3DB
Re: Limited security support for Go/Rust? Re ssh3
On 1/16/24 13:56, Jérémy Lal wrote: As Built-Using is for license compliance only, no? See https://www.debian.org/doc/debian-policy/ch-relationships.html#additional-source-packages-used-to-build-the-binary-built-using Indeed, thanks for the link. it seems that many people think that "Built-Using" can be used to express static linking (including yours truly, even though i *know* that it is meant for license compliance only). which makes me wonder: probably we should have an additional field that expresses such static linking (and therefore would trigger a rebuild when the dependency changes). or we could finally accept that many¹ people would just use "Built-Using" for this anyhow, and explicitly allow such use. dmas IOhannes ¹ i have the gut feeling i am not alone :-) but i don't have any numbers OpenPGP_0xB65019C47F7A36F8.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Limited security support for Go/Rust? Re ssh3
Le mar. 16 janv. 2024 à 13:09, Bastian Blank a écrit : > On Tue, Jan 16, 2024 at 11:22:48AM +0100, Jérémy Lal wrote: > > I naively believed that golang-* packages expressed those dependencies > with > > "Built-Using". > > As Built-Using is for license compliance only, no? > > See > > https://www.debian.org/doc/debian-policy/ch-relationships.html#additional-source-packages-used-to-build-the-binary-built-using Indeed, thanks for the link.
Re: Limited security support for Go/Rust? Re ssh3
On Tue, Jan 16, 2024 at 10:59:30AM +0100, Simon Josefsson wrote: > Rebuilding a bit more than what is strictly needed sounds fine as a > first solution to me. Building maybe. But how do you want to publish them? The security archive is not made to handle that. > My naive approach on how to fix a security problem in package X which is > statically embedded into other packages A, B, C, ... would be to rebuild > the transitive closure of all packages that Build-Depends on X and > publish a security update for all those packages. So if a fix to the net/tls module of go shows up (happens from time to time), all go packages needs to be rebuilt? Bastian -- Many Myths are based on truth -- Spock, "The Way to Eden", stardate 5832.3
Re: Limited security support for Go/Rust? Re ssh3
On Tue, Jan 16, 2024 at 11:22:48AM +0100, Jérémy Lal wrote: > I naively believed that golang-* packages expressed those dependencies with > "Built-Using". As Built-Using is for license compliance only, no? See https://www.debian.org/doc/debian-policy/ch-relationships.html#additional-source-packages-used-to-build-the-binary-built-using Bastian -- "Life and death are seldom logical." "But attaining a desired goal always is." -- McCoy and Spock, "The Galileo Seven", stardate 2821.7
Re: Limited security support for Go/Rust? Re ssh3
tis 2024-01-16 klockan 11:22 +0100 skrev Jérémy Lal: > > > Le mar. 16 janv. 2024 à 11:00, Simon Josefsson > a écrit : > > Paul Wise writes: > > > > > On Mon, 2024-01-15 at 10:17 +0100, Bastian Blank wrote: > > > > > > > I asked for practical solutions, not theoretical ones. We > > > > don't have a > > > > suitable way to rebuild all packages just because right now. > > > > > > There are some ideas on the static linking wiki page: > > > > > > https://wiki.debian.org/StaticLinking > > > > > > Probably the most practical solution for today would be to use a > > > build > > > info database to find out which builds had installed binary > > > packages > > > containing insecure statically linkable files of any kind, then > > > rebuild > > > the source packages that were affected. There is a 2019 demo > > > here: > > > > > > https://salsa.debian.org/bremner/builtin-pho/-/blob/master/demos/needs-rebuild.sh > > > https://www.cs.unb.ca/~bremner//blog/posts/builtin-pho/ > > > > > > This may mean rebuilding more packages than were really needed, > > > but a more exact method would require full tracing of input data > > > to > > > output data during builds being added to all toolchains, which > > > seems > > > like a much longer term project than buildinfo based rebuilds. > > > > Rebuilding a bit more than what is strictly needed sounds fine as a > > first solution to me. > > > > My naive approach on how to fix a security problem in package X > > which is > > statically embedded into other packages A, B, C, ... would be to > > rebuild > > the transitive closure of all packages that Build-Depends on X and > > publish a security update for all those packages. > > > > What is the problem with that approach to handle security problems > > in a > > Go package for trixie? > > > > I'm sure the number of packages to rebuild could be reduced in > > various > > clever ways, but until we have tooling to automate that, a naive > > but > > costly approach seems feasible, unless i'm missing something. > > > > How large is the gap between tracing buildinfo information and > > simply > > relying on Build-Depends? > > > > Isn't the gap between using Build-Depends and the buildinfo- > > approach > > only concerning the always-assumed-to-be-installed packages like > > libc or > > /bin/sh which I never seems to recall if they are what build- > > essential > > installs, or what the policy manual says it should do, or what > > 'Essential: yes' implies, or 'Priority: required' implies, etc. > > For Go > > packages I don't think they are relevant in practice. > > > > > I naively believed that golang-* packages expressed those > dependencies with "Built-Using". True, but I was thinking of a solution that would not be Go-specific. I realized that there is one problem with my approach: consider if package A was built via Build-Depends package B of version X and that later package B is upgraded to X+1 in Debian. Then if a security problem happens in B we need to rebuild A. It may be that package A no longer builds due to some incompatibility between version X and X+1 of B. This would not be noticed until a full rebuild of an archive is done, or when the security teams wants to rebuild the transitive closure of the Build-Depends graph for a package. I still think this is something we just need to be prepared to handle, by patching packages to fix the build problem in whatever way is appropriate. It will require some more patching to really fix the security problem, and allow packages to be rebuilt with the new version. /Simon signature.asc Description: This is a digitally signed message part
Re: Limited security support for Go/Rust? Re ssh3
Le mar. 16 janv. 2024 à 11:00, Simon Josefsson a écrit : > Paul Wise writes: > > > On Mon, 2024-01-15 at 10:17 +0100, Bastian Blank wrote: > > > >> I asked for practical solutions, not theoretical ones. We don't have a > >> suitable way to rebuild all packages just because right now. > > > > There are some ideas on the static linking wiki page: > > > > https://wiki.debian.org/StaticLinking > > > > Probably the most practical solution for today would be to use a build > > info database to find out which builds had installed binary packages > > containing insecure statically linkable files of any kind, then rebuild > > the source packages that were affected. There is a 2019 demo here: > > > > > https://salsa.debian.org/bremner/builtin-pho/-/blob/master/demos/needs-rebuild.sh > > https://www.cs.unb.ca/~bremner//blog/posts/builtin-pho/ > > > > This may mean rebuilding more packages than were really needed, > > but a more exact method would require full tracing of input data to > > output data during builds being added to all toolchains, which seems > > like a much longer term project than buildinfo based rebuilds. > > Rebuilding a bit more than what is strictly needed sounds fine as a > first solution to me. > > My naive approach on how to fix a security problem in package X which is > statically embedded into other packages A, B, C, ... would be to rebuild > the transitive closure of all packages that Build-Depends on X and > publish a security update for all those packages. > > What is the problem with that approach to handle security problems in a > Go package for trixie? > > I'm sure the number of packages to rebuild could be reduced in various > clever ways, but until we have tooling to automate that, a naive but > costly approach seems feasible, unless i'm missing something. > > How large is the gap between tracing buildinfo information and simply > relying on Build-Depends? > > Isn't the gap between using Build-Depends and the buildinfo-approach > only concerning the always-assumed-to-be-installed packages like libc or > /bin/sh which I never seems to recall if they are what build-essential > installs, or what the policy manual says it should do, or what > 'Essential: yes' implies, or 'Priority: required' implies, etc. For Go > packages I don't think they are relevant in practice. > I naively believed that golang-* packages expressed those dependencies with "Built-Using". Jérémy
Re: Limited security support for Go/Rust? Re ssh3
Paul Wise writes: > On Mon, 2024-01-15 at 10:17 +0100, Bastian Blank wrote: > >> I asked for practical solutions, not theoretical ones. We don't have a >> suitable way to rebuild all packages just because right now. > > There are some ideas on the static linking wiki page: > > https://wiki.debian.org/StaticLinking > > Probably the most practical solution for today would be to use a build > info database to find out which builds had installed binary packages > containing insecure statically linkable files of any kind, then rebuild > the source packages that were affected. There is a 2019 demo here: > > https://salsa.debian.org/bremner/builtin-pho/-/blob/master/demos/needs-rebuild.sh > https://www.cs.unb.ca/~bremner//blog/posts/builtin-pho/ > > This may mean rebuilding more packages than were really needed, > but a more exact method would require full tracing of input data to > output data during builds being added to all toolchains, which seems > like a much longer term project than buildinfo based rebuilds. Rebuilding a bit more than what is strictly needed sounds fine as a first solution to me. My naive approach on how to fix a security problem in package X which is statically embedded into other packages A, B, C, ... would be to rebuild the transitive closure of all packages that Build-Depends on X and publish a security update for all those packages. What is the problem with that approach to handle security problems in a Go package for trixie? I'm sure the number of packages to rebuild could be reduced in various clever ways, but until we have tooling to automate that, a naive but costly approach seems feasible, unless i'm missing something. How large is the gap between tracing buildinfo information and simply relying on Build-Depends? Isn't the gap between using Build-Depends and the buildinfo-approach only concerning the always-assumed-to-be-installed packages like libc or /bin/sh which I never seems to recall if they are what build-essential installs, or what the policy manual says it should do, or what 'Essential: yes' implies, or 'Priority: required' implies, etc. For Go packages I don't think they are relevant in practice. /Simon signature.asc Description: PGP signature
Re: Limited security support for Go/Rust? Re ssh3
Is rebuilding really the biggest problem? Even if Debian had enough capacity to rebuild everything after the change of a build dependency, I do not see how this solves the work of tracking Rust dependencies in the first place. I use a handful of Rust applications which I am building myself on Debian. Each of them has hundreds of tiny dependencies which are changing on a daily (if not hourly) basis. Some of them I can only build in Sid, or I have to checkout really old commits from Git. I do not see how this is supposed to be working in a Debian stable release with all the required manual work such as backporting security fixes etc. Regards Stephan signature.asc Description: This is a digitally signed message part
Re: Limited security support for Go/Rust? Re ssh3
On Mon, 2024-01-15 at 10:17 +0100, Bastian Blank wrote: > I asked for practical solutions, not theoretical ones. We don't have a > suitable way to rebuild all packages just because right now. There are some ideas on the static linking wiki page: https://wiki.debian.org/StaticLinking Probably the most practical solution for today would be to use a build info database to find out which builds had installed binary packages containing insecure statically linkable files of any kind, then rebuild the source packages that were affected. There is a 2019 demo here: https://salsa.debian.org/bremner/builtin-pho/-/blob/master/demos/needs-rebuild.sh https://www.cs.unb.ca/~bremner//blog/posts/builtin-pho/ This may mean rebuilding more packages than were really needed, but a more exact method would require full tracing of input data to output data during builds being added to all toolchains, which seems like a much longer term project than buildinfo based rebuilds. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Limited security support for Go/Rust? Re ssh3
On Sun, Jan 14, 2024 at 04:24:57PM +0100, Simon Josefsson wrote: > Isn't that what the text refers to? Vendoring and static linking are > two examples of the same problem that the security team may encounter. We accept vendoring of autoconf/automake/gnulib distro wide. Please show practical problems with it? The result of autoconf and automake is not embedded into the produced binaries, so a security vulnerability can not go on. The vendoring of gnulib, well, is old and maybe you could show that it is a problem in the sources that have it, aka they don't handle security fixes and at the same time don't change the library. > The problem with dependencies are more obvious for Go/Rust code but I > think we always have had that problem anyway. Here the problem is embedded into the language oekosystem itself. It is not a choice of the software author to do static linking. > As for solutions, isn't the solution to both vendoring or statical > linking the obvious one? You will have to rebuild all packages that > contain the security vulnerability. I asked for practical solutions, not theoretical ones. We don't have a suitable way to rebuild all packages just because right now. > Maybe I'm missing how these two problems result in different problems > for the security team? You did not show that autoconf/automak/gnulib is a problem at all. And a => b with a false gives no answer. Bastian -- Spock: The odds of surviving another attack are 13562190123 to 1, Captain.
Re: Limited security support for Go/Rust? Re ssh3
Simon Josefsson wrote: > Isn't that what the text refers to? Vendoring and static linking are > two examples of the same problem that the security team may encounter. > The problem with dependencies are more obvious for Go/Rust code but I > think we always have had that problem anyway. Another example of this class of problem is header only C++ libraries. -- Robert Edmonds edmo...@debian.org
Re: Limited security support for Go/Rust? Re ssh3
Bastian Blank writes: > Hi Simon > > On Sun, Jan 14, 2024 at 10:47:18AM +0100, Simon Josefsson wrote: >> As an analogy, consider the ./configure scripts that is generated by >> autoconf during build of many packages. The script typically generate >> code that is put into config.h that is used (statically) during >> compilation of the binaries that are shipped by Debian. > > Could you show an example, where there is actually code injcted in this > stage? And then, this is vendoring, not static linking. > >> You could also compare how the source-level reuse-library gnulib is used >> by many essential packages (coreutils, grep, sed, awk, tar, etc), with >> large code-reuse that influences the installed binaries. A security >> sensitive bug in gnulib would require rebuild of many packages. > > That is not static linking, this is vendoring. And can you show that > GNU utils don't fix security bugs on this vendored lib? > >> My suggestion is that we relax or remove the Go/Rust statement in future >> release notes. > > No. You described completely different circumstances. > > Or do you have a practical solution for the static linking problem, not > the vendoring problem that you actually compared it against? Right, these are slightly different technical problems, but as far as the brief discussion in the release notes -- https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html#golang-static-linking -- I think the relevant aspect is identical: package X in Debian contains an embedded copy of code whose corresponding source code is in package Y in Debian, and fixing a security problem in Y will require rebuilding X and the entire dependency chain between X and Y that carry the code that ends up in X. Isn't that what the text refers to? Vendoring and static linking are two examples of the same problem that the security team may encounter. The problem with dependencies are more obvious for Go/Rust code but I think we always have had that problem anyway. As for solutions, isn't the solution to both vendoring or statical linking the obvious one? You will have to rebuild all packages that contain the security vulnerability. Maybe I'm missing how these two problems result in different problems for the security team? /Simon signature.asc Description: PGP signature
Re: Limited security support for Go/Rust? Re ssh3
Hi Simon On Sun, Jan 14, 2024 at 10:47:18AM +0100, Simon Josefsson wrote: > As an analogy, consider the ./configure scripts that is generated by > autoconf during build of many packages. The script typically generate > code that is put into config.h that is used (statically) during > compilation of the binaries that are shipped by Debian. Could you show an example, where there is actually code injcted in this stage? And then, this is vendoring, not static linking. > You could also compare how the source-level reuse-library gnulib is used > by many essential packages (coreutils, grep, sed, awk, tar, etc), with > large code-reuse that influences the installed binaries. A security > sensitive bug in gnulib would require rebuild of many packages. That is not static linking, this is vendoring. And can you show that GNU utils don't fix security bugs on this vendored lib? > My suggestion is that we relax or remove the Go/Rust statement in future > release notes. No. You described completely different circumstances. Or do you have a practical solution for the static linking problem, not the vendoring problem that you actually compared it against? Bastian -- A father doesn't destroy his children. -- Lt. Carolyn Palamas, "Who Mourns for Adonais?", stardate 3468.1.
Re: Limited security support for Go/Rust? Re ssh3
On Sun, 2024-01-14 at 10:47 +0100, Simon Josefsson wrote: > The more I think about it, I think it seems unfair to categorize this > as a Go/Rust problem. The point is that it should be possible all packages in Debian without dependencies which are outside of Debian. The same problem exists with all programs that depend on external dependency managers which manage a large number of small libraries which are changing all the time. These dependencies are a moving target hard to include into Debian stable. This is not only making updates but also reproducible builds harder. More examples for this are Node NPM, Python PIP, Ruby Gems etc. Go and Rust are different than NPM and Python for two reasons: - Go and Rust are used for more low-level programs where stability matters even more. - Go and Rust use static linking. > Interesting -- what is the current thinking about this problem? I think the current plan is to automatically update Debian packages of Go/Rust libraries in a similar way as it is done for Python. However, the static linking makes this different from Python, as these packages are build dependencies rather than runtime dependencies. So all reverse dependencies have to be rebuild in some way. I do not know what the status/progress of this project is. Regards Stephan signature.asc Description: This is a digitally signed message part
Re: Limited security support for Go/Rust? Re ssh3
On Sun, Jan 14, 2024 at 10:47:18AM +0100, Simon Josefsson wrote: > Stephan Verbücheln writes: > > > On Sat, 30 Dec 2023 12:47:48 + Colin Watson > > wrote: > >> I also feel that something security-critical like this that's > >> labelled by upstream as "still experimental" probably shouldn't > >> be in a Debian release. > > > > It is written in Go. The problem of Go library support in Debian should > > also be considered for a security-critical tool like this. > > > > https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html#golang-static-linking > > Interesting -- what is the current thinking about this problem? > > The more I think about it, I think it seems unfair to categorize this as > a Go/Rust problem. +1. Container packages such as docker and podman are also golang packages and also security critical. > ... > My suggestion is that we relax or remove the Go/Rust statement in future > release notes. Or we could as well look at improving the infrastructure to deal with them. Best, Nilesh signature.asc Description: PGP signature
