Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Wed, Jan 24, 2024 at 09:37:27AM +0100, Simon Josefsson wrote: > Simon Josefsson writes: > > >> > My naive approach on how to fix a security problem in package X > >> > which is > >> > statically embedded into other packages A, B, C, ... would be to > >> > rebuild > >> > the transitive closure of all packages that Build-Depends on X and > >> > publish a security update for all those packages. > ... > > I realized that there is one problem with my approach: consider if > > package A was built via Build-Depends package B of version X and that > > later package B is upgraded to X+1 in Debian. Then if a security > > problem happens in B we need to rebuild A. It may be that package A no > > longer builds due to some incompatibility between version X and X+1 of > > B. This would not be noticed until a full rebuild of an archive is > > done, or when the security teams wants to rebuild the transitive > > closure of the Build-Depends graph for a package. > > Having reflected a bit, and learned through my own experience and > others' insights [1] that Go Build-Depends are not transitive, I'd like > to update my proposal on how to handle a security bug in any Go/Rust/etc > package and the resulting package rebuilds: > > To fix a security problem in package X, which is used during build > (through statical linking, vendoring, or some other mechanism) to > build package A, B, C, ... you need to rebuild all packages that > Build-Depends on X (lets call this step 1) and all packages that > Build-Depends on any of the packages that will be rebuilt during step > 1. This rebuild process is iterated until no more packages remains to > be rebuild, and all packages have been rebuilt using newly rebuild > packages. If there are cyclical dependencies (which unfortunately are > common), you have to loop until all packages have been rebuilt with a > clean dependency chain, and detect the loop and stop rebuilding. If > there are FTBFS errors during the rebuilds, this will have to be > patched too. > > Yes, for a low-level Go package (e.g., golang-golang-x-net-dev), this > will mean rebuilding almost all of the Go packages in Debian and publish > them in a security advisory. > > This algorithm can be optimized (i.e., reduce the number of packages to > publish in an advisory) by either of: > > 1) using information from Built-Using: (which was not designed for >this purpose, so this is fragile) or *.buildinfo. > > 2) by dropping all 'Architecture: all' packages that does not embedd >the buggy code. > > The last optimization 2) would reduce the number of Go packages to > publish significantly, as it would drop most golang-*-dev packages. I > think this actually makes this process feasible in practice, as there > are relatively few binary packages written in Go. > > This method applies to non-Go/Rust too, if there are such security > problems and reverse build chains. > > I think all of this (except maybe the optimization 2) which requires > code comparisons) can be automated in a GitLab pipeline for higher > confidence of the result. Here's some rough thoughts and numbers for the (packaged) Rust ecosystem.. Currently in unstable we have ~100 packages shipping binaries written in Rust that are packaged using the "stock" packaging schema (that's debcargo/debcargo-conf + dh-cargo + our cargo wrapper) or Jonas' forked one (which use a slightly different, but mostly compatible approach), which means dh-cargo-built-using is called and encodes the static linking information in the binary package in X-Cargo-Built-Using[0]. Let's add another 50[1] that are using Rust in some fashion and thus (potentially) contain statically linked Rust code from other source packages, which are for whatever reason currently not emitting this information - some examples: src:cargo, src:rustc - these both vendor their deps and need special handling anyhow, firefox and soon chromium (similar AFAIK), but also more relevant, python3-cryptography and friends, or parts of Gnome that are newly or re-written in Rust (these could and should emit the information, but currently don't). These are the only packages (besides the librust-foo-dev package triggering a rebuild) that need a rebuild right now if a security fix needs to be deployed. But thankfully we don't need to rebuild 1xx Rust-using packages whenever any crate has a security fix, we only need to do so if they transitively build-depend on the vulnerable (now fixed) package/crate. This in effect also means that (for simple cases where the vulnerability is not spread across multiple crates) we can rebuild the vulnerable (lib) crate first, and then schedule independent rebuilds of all binary-shipping packages that statically link it. I attached a list doing a rough mapping of source package to number of such packages needing a rebuild if the source package in question gets a security fix. In my experience (both packaging Rust things in Debian,
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Jan 24, Peter Pentchev wrote:
> This might be a minority, optimistic, rose-tinted-glasses kind of
> opinion, but I believe that the state of the Rust ecosystem today
> (I have no experience with the Go one) is quite similar to what Perl and
> Python modules were 25, 20, bah, even 15 years ago. Gradually, with time,
I am not familiar with the Python ecosystem, but I have been writing
Perl and packaging software with Perl dependencies for over 25 years and
I can confidently say that this is not true.
Perl libraries ("modules") generally never had the API instability that
I have seen in Rust libraries (but much less in Go, I believe).
In my experience forward compatibility has always been very important in
the Perl ecosystem.
BTW, for the past couple of years I have been presenting to my other
community, the network operators, about some of my Debian work and these
problems with integrating complex Rust software in distributions, e.g.
https://www.linux.it/~md/text/rpki-validators-euroix2023.pdf .
--
ciao,
Marco
signature.asc
Description: PGP signature
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Wed, Jan 24, 2024 at 01:01:34PM +, Luca Boccassi wrote: > On Wed, 24 Jan 2024 at 12:26, Johannes Schauer Marin Rodrigues > wrote: > > > > Hi, > > > > Quoting Luca Boccassi (2024-01-24 12:59:38) > > > There's always option B: recognize that the Rust/Go ecosystems are not > > > designed to be compatible with the Linux distributions model, and are > > > instead > > > designed to be as convenient as possible for a _single_ application > > > developer > > > and its users - at the detriment of everybody else - and for large > > > corporations that ship a handful of applications with swathes of engineers > > > that can manage the churn, and it is not made nor intended to scale to > > > ~60k > > > packages or whatever number we have today in unstable. And simply stop > > > attempting to merge these two incompatible ecosystems against their will, > > > at > > > the very least until and unless they reach feature and stability parity > > > with > > > the C/C++ ecosystems - stable API/ABI and dynamic linking by default. > > > > > > There are many ways to ship applications today that are much better > > > suited for these models, like Flatpak, where the maintenance burden is > > > shifted onto those who choose to opt in to such ecosystems. > > > > how does that work for those applications that require rust, go and friends? > > Are you proposing that everything that needs them should be be distributed > > by a > > flatpak or similar mechanism instead? > > Those applications are not shipped in the distribution. If somebody > wants to use them, they'll have to figure it out, just like for > everything else that is not shipped in the distribution, which is > already a subset of all available software in the world. > > > Just a few days ago I tried building mesa from experimental (otherwise there > > are severe graphics glitches on my platform) on bookworm and everything > > worked > > except its rust build dependencies. I had no luck trying to backport those > > parts of rust that I needed and even if it had worked, it would've meant > > backporting dozens of rust packages just to have a backport of mesa. It > > seemed > > way simpler to just disable the mesa component that requires rust and call > > it a > > day. But that probably doesn't work for all applications and maybe sometimes > > the component written in rust is actually what you want. And in case of > > mesa, a > > flatpak of it probably would also not've helped as that would've meant that > > I > > need to run everything that I want to run with a more modern mesa based on > > that > > flatpak, right? > > > > So how is option B a solution in practice? > > You have found first-hand the exact problem with this ecosystem. Now > try to imagine doing that for 30k packages, all vendoring, pinning and > static linking against each other at different, incompatible versions. > Again, the solution in practice is to simply disable or patch out > those components, and if somebody needs them, they can complain to > upstream and ask them for a solution, if there is one. If there isn't, > though luck. And yes, that is not great - but neither are all other > options, so it seems much wiser to me to push responsibility where it > belongs - with the upstreams choosing to use such ecosystems, and the > owner of said ecosystems choosing to make them incompatible with the > distribution model, as they are in the best position to solve the > problem(s) that happen due to their design choices. This might be a minority, optimistic, rose-tinted-glasses kind of opinion, but I believe that the state of the Rust ecosystem today (I have no experience with the Go one) is quite similar to what Perl and Python modules were 25, 20, bah, even 15 years ago. Gradually, with time, the module authors realized that if they wanted people to use their modules, they would eventually have to settle on a stable API and only make incompatible changes very rarely. With time, with this realization trickling up and down across the most used libraries, things slowly start to get better. Yes, even now there are what can only be called "transitions" in the Perl and Python Debian packaging - some often-used module makes an incompatible change and the packagers need to wait until all the other packaged modules have either caught up or it has somehow been proven through testing that some of the dependent modules are not really affected by the incompatible change. In my experience, nowadays this happens much, much more rarely than 10 or 15 years ago. I have no idea how long it will take the Rust ecosystem to realize that. I know that some of the most widely used modules have already done that, some of them have been at the same 0.x.* or even 1.* version for years (yes, really, years). So... here's hoping. G'luck, Peter -- Peter Pentchev r...@ringlet.net r...@debian.org p...@storpool.com PGP key:http://people.FreeBSD.org/~roam/roam.key.asc Key fingerprint 2EE7 A7A5 17FC 124C
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On 2024-01-24 13:26:49 +0100 (+0100), Johannes Schauer Marin Rodrigues wrote: [...] > how does that work for those applications that require rust, go > and friends? Are you proposing that everything that needs them > should be be distributed by a flatpak or similar mechanism > instead? > > Just a few days ago I tried building mesa from experimental > (otherwise there are severe graphics glitches on my platform) on > bookworm and everything worked except its rust build dependencies. > I had no luck trying to backport those parts of rust that I needed > and even if it had worked, it would've meant backporting dozens of > rust packages just to have a backport of mesa. [...] Another practical example is Python applications/libs increasingly integrating performance or security critical routines (re)written in Rust. The python-cryptography source package in sid build-depends on cargo and a dozen Rust libs. I didn't bother to pull a rdeps list for it, but can guarantee a vast swath of the python3-* packages in Debian depend on python3-cryptography, and this particular problem will only get worse as the CPython core devs see the rising popularity of the externally-developed cryptography library as a good reason to strip any remnants of cryptographic modules and bindings from the stdlib. -- Jeremy Stanley signature.asc Description: PGP signature
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Wed, 24 Jan 2024 at 12:26, Johannes Schauer Marin Rodrigues wrote: > > Hi, > > Quoting Luca Boccassi (2024-01-24 12:59:38) > > There's always option B: recognize that the Rust/Go ecosystems are not > > designed to be compatible with the Linux distributions model, and are > > instead > > designed to be as convenient as possible for a _single_ application > > developer > > and its users - at the detriment of everybody else - and for large > > corporations that ship a handful of applications with swathes of engineers > > that can manage the churn, and it is not made nor intended to scale to ~60k > > packages or whatever number we have today in unstable. And simply stop > > attempting to merge these two incompatible ecosystems against their will, at > > the very least until and unless they reach feature and stability parity with > > the C/C++ ecosystems - stable API/ABI and dynamic linking by default. > > > > There are many ways to ship applications today that are much better > > suited for these models, like Flatpak, where the maintenance burden is > > shifted onto those who choose to opt in to such ecosystems. > > how does that work for those applications that require rust, go and friends? > Are you proposing that everything that needs them should be be distributed by > a > flatpak or similar mechanism instead? Those applications are not shipped in the distribution. If somebody wants to use them, they'll have to figure it out, just like for everything else that is not shipped in the distribution, which is already a subset of all available software in the world. > Just a few days ago I tried building mesa from experimental (otherwise there > are severe graphics glitches on my platform) on bookworm and everything worked > except its rust build dependencies. I had no luck trying to backport those > parts of rust that I needed and even if it had worked, it would've meant > backporting dozens of rust packages just to have a backport of mesa. It seemed > way simpler to just disable the mesa component that requires rust and call it > a > day. But that probably doesn't work for all applications and maybe sometimes > the component written in rust is actually what you want. And in case of mesa, > a > flatpak of it probably would also not've helped as that would've meant that I > need to run everything that I want to run with a more modern mesa based on > that > flatpak, right? > > So how is option B a solution in practice? You have found first-hand the exact problem with this ecosystem. Now try to imagine doing that for 30k packages, all vendoring, pinning and static linking against each other at different, incompatible versions. Again, the solution in practice is to simply disable or patch out those components, and if somebody needs them, they can complain to upstream and ask them for a solution, if there is one. If there isn't, though luck. And yes, that is not great - but neither are all other options, so it seems much wiser to me to push responsibility where it belongs - with the upstreams choosing to use such ecosystems, and the owner of said ecosystems choosing to make them incompatible with the distribution model, as they are in the best position to solve the problem(s) that happen due to their design choices.
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
Hi, Quoting Luca Boccassi (2024-01-24 12:59:38) > There's always option B: recognize that the Rust/Go ecosystems are not > designed to be compatible with the Linux distributions model, and are instead > designed to be as convenient as possible for a _single_ application developer > and its users - at the detriment of everybody else - and for large > corporations that ship a handful of applications with swathes of engineers > that can manage the churn, and it is not made nor intended to scale to ~60k > packages or whatever number we have today in unstable. And simply stop > attempting to merge these two incompatible ecosystems against their will, at > the very least until and unless they reach feature and stability parity with > the C/C++ ecosystems - stable API/ABI and dynamic linking by default. > > There are many ways to ship applications today that are much better > suited for these models, like Flatpak, where the maintenance burden is > shifted onto those who choose to opt in to such ecosystems. how does that work for those applications that require rust, go and friends? Are you proposing that everything that needs them should be be distributed by a flatpak or similar mechanism instead? Just a few days ago I tried building mesa from experimental (otherwise there are severe graphics glitches on my platform) on bookworm and everything worked except its rust build dependencies. I had no luck trying to backport those parts of rust that I needed and even if it had worked, it would've meant backporting dozens of rust packages just to have a backport of mesa. It seemed way simpler to just disable the mesa component that requires rust and call it a day. But that probably doesn't work for all applications and maybe sometimes the component written in rust is actually what you want. And in case of mesa, a flatpak of it probably would also not've helped as that would've meant that I need to run everything that I want to run with a more modern mesa based on that flatpak, right? So how is option B a solution in practice? Thanks! cheers, josch signature.asc Description: signature
Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)
On Wed, 24 Jan 2024 at 11:42, Simon Josefsson wrote: > > Simon Josefsson writes: > > >> > My naive approach on how to fix a security problem in package X > >> > which is > >> > statically embedded into other packages A, B, C, ... would be to > >> > rebuild > >> > the transitive closure of all packages that Build-Depends on X and > >> > publish a security update for all those packages. > ... > > I realized that there is one problem with my approach: consider if > > package A was built via Build-Depends package B of version X and that > > later package B is upgraded to X+1 in Debian. Then if a security > > problem happens in B we need to rebuild A. It may be that package A no > > longer builds due to some incompatibility between version X and X+1 of > > B. This would not be noticed until a full rebuild of an archive is > > done, or when the security teams wants to rebuild the transitive > > closure of the Build-Depends graph for a package. > > Having reflected a bit, and learned through my own experience and > others' insights [1] that Go Build-Depends are not transitive, I'd like > to update my proposal on how to handle a security bug in any Go/Rust/etc > package and the resulting package rebuilds: There's always option B: recognize that the Rust/Go ecosystems are not designed to be compatible with the Linux distributions model, and are instead designed to be as convenient as possible for a _single_ application developer and its users - at the detriment of everybody else - and for large corporations that ship a handful of applications with swathes of engineers that can manage the churn, and it is not made nor intended to scale to ~60k packages or whatever number we have today in unstable. And simply stop attempting to merge these two incompatible ecosystems against their will, at the very least until and unless they reach feature and stability parity with the C/C++ ecosystems - stable API/ABI and dynamic linking by default. There are many ways to ship applications today that are much better suited for these models, like Flatpak, where the maintenance burden is shifted onto those who choose to opt in to such ecosystems.
