Re: Proposal for how to deal with Go/Rust/etc security bugs

[Philipp Kern]

On 2024-01-25 17:10, Russ Allbery wrote:

Rebuilding a bunch of software after a security fix is not a completely
intractable problem that we have no idea how to even approach.  It's 
just

CPU cycles and good metadata plus ensuring that our software can be
rebuilt, something that we already promise.  Some aspects of making 
this

work will doubtless be *annoying*, but it doesn't seem outside of our
capabilities as a project.


One worry I'd have is that we have - in the past - been very 
conservative in what we rebuilt. We have not rebuilt the archive 
pre-release either (maybe we should!). So you are suddenly rebuilding 
binaries with a new toolchain and updated inputs - and while it's easy 
to review source diffs, there are always some unknowns what is happening 
to the binary as a result. From a(n ex) release perspective that would 
be my main worry.


I agree that the rebuilds themselves are a matter of programming, 
assuming they succeed and you don't need to do them in stages to resolve 
dependencies. Presumably we'd then need a pipeline to publish partial 
security updates as the leaves of the tree complete (or enough of a 
heads-up). If you have stages because intermediate builds incorporate 
bits of other packages and re-export them into build environments 
(unlikely?) or if you need to shepherd a lot of failed builds and try to 
debug what happened, then it becomes a lot more toilsome and 
labor-intensive.


Kind regards
Philipp Kern

Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)

[Fabian Grünbichler]

On Wed, Jan 24, 2024 at 09:37:27AM +0100, Simon Josefsson wrote:
> Simon Josefsson  writes:
> 
> >> > My naive approach on how to fix a security problem in package X
> >> > which is
> >> > statically embedded into other packages A, B, C, ... would be to
> >> > rebuild
> >> > the transitive closure of all packages that Build-Depends on X and
> >> > publish a security update for all those packages.
> ...
> > I realized that there is one problem with my approach: consider if
> > package A was built via Build-Depends package B of version X and that
> > later package B is upgraded to X+1 in Debian.  Then if a security
> > problem happens in B we need to rebuild A. It may be that package A no
> > longer builds due to some incompatibility between version X and X+1 of
> > B.  This would not be noticed until a full rebuild of an archive is
> > done, or when the security teams wants to rebuild the transitive
> > closure of the Build-Depends graph for a package.
> 
> Having reflected a bit, and learned through my own experience and
> others' insights [1] that Go Build-Depends are not transitive, I'd like
> to update my proposal on how to handle a security bug in any Go/Rust/etc
> package and the resulting package rebuilds:
> 
>   To fix a security problem in package X, which is used during build
>   (through statical linking, vendoring, or some other mechanism) to
>   build package A, B, C, ... you need to rebuild all packages that
>   Build-Depends on X (lets call this step 1) and all packages that
>   Build-Depends on any of the packages that will be rebuilt during step
>   1.  This rebuild process is iterated until no more packages remains to
>   be rebuild, and all packages have been rebuilt using newly rebuild
>   packages.  If there are cyclical dependencies (which unfortunately are
>   common), you have to loop until all packages have been rebuilt with a
>   clean dependency chain, and detect the loop and stop rebuilding.  If
>   there are FTBFS errors during the rebuilds, this will have to be
>   patched too.
> 
> Yes, for a low-level Go package (e.g., golang-golang-x-net-dev), this
> will mean rebuilding almost all of the Go packages in Debian and publish
> them in a security advisory.
> 
> This algorithm can be optimized (i.e., reduce the number of packages to
> publish in an advisory) by either of:
> 
> 1) using information from Built-Using: (which was not designed for
>this purpose, so this is fragile) or *.buildinfo.
> 
> 2) by dropping all 'Architecture: all' packages that does not embedd
>the buggy code.
> 
> The last optimization 2) would reduce the number of Go packages to
> publish significantly, as it would drop most golang-*-dev packages.  I
> think this actually makes this process feasible in practice, as there
> are relatively few binary packages written in Go.
> 
> This method applies to non-Go/Rust too, if there are such security
> problems and reverse build chains.
> 
> I think all of this (except maybe the optimization 2) which requires
> code comparisons) can be automated in a GitLab pipeline for higher
> confidence of the result.

Here's some rough thoughts and numbers for the (packaged) Rust
ecosystem..

Currently in unstable we have ~100 packages shipping binaries
written in Rust that are packaged using the "stock" packaging schema
(that's debcargo/debcargo-conf + dh-cargo + our cargo wrapper) or Jonas'
forked one (which use a slightly different, but mostly compatible
approach), which means dh-cargo-built-using is called and encodes the
static linking information in the binary package in
X-Cargo-Built-Using[0].

Let's add another 50[1] that are using Rust in some fashion and thus
(potentially) contain statically linked Rust code from other source
packages, which are for whatever reason currently not emitting this
information - some examples: src:cargo, src:rustc - these both vendor
their deps and need special handling anyhow, firefox and soon chromium
(similar AFAIK), but also more relevant, python3-cryptography and
friends, or parts of Gnome that are newly or re-written in Rust (these
could and should emit the information, but currently don't).

These are the only packages (besides the librust-foo-dev package
triggering a rebuild) that need a rebuild right now if a security fix
needs to be deployed. But thankfully we don't need to rebuild 1xx
Rust-using packages whenever any crate has a security fix, we only need
to do so if they transitively build-depend on the vulnerable (now fixed)
package/crate. This in effect also means that (for simple cases where
the vulnerability is not spread across multiple crates) we can rebuild
the vulnerable (lib) crate first, and then schedule independent rebuilds
of all binary-shipping packages that statically link it. I attached a
list doing a rough mapping of source package to number of such packages
needing a rebuild if the source package in question gets a security fix.
In my experience (both packaging Rust things in Debian, 

Re: Understanding what's missing for Rust dynamic linking (was: Proposal for how to deal with Go/Rust/etc security bugs)

[Fabian Grünbichler]

On Thu, Jan 25, 2024 at 07:03:05PM +, Luca Boccassi wrote:
> On Thu, 25 Jan 2024 at 18:22, Gard Spreemann  wrote:
> >
> > Hello.
> >
> > Paul Wise  writes:
> >
> > > On Thu, 2024-01-25 at 00:24 +, Wookey wrote:
> > >
> > >> People keep telling us (@ARM) how marvellous Rust is, and we keep
> > >> telling them that it's useless in the real world until it sorts out
> > >> the stable ABI/dynamic linking problem.
> > >
> > > IIRC that has been worked on for some years now, and IIRC
> > > the static linking wiki page has some references about this.
> > >
> > > https://wiki.debian.org/StaticLinking
> >
> > This reminded me that I'm not even sure that I fully understand what
> > Rust's remaining technical obstacles to achieving dynamic linking (at
> > least within Debian) are. I'm ignoring the potential cultural or
> > political issues that have been alluded to by others. My understanding –
> > and please do correct me! – has been that three components are missing:
> >
> > (1) A stable ABI.
> <...>
> > From Debian's perspective, is really (1) all that important given that a
> > stable release only has to deal with a specific version of the compiler?
> > Could we not live with every new version of *just* rustc in sid
> > introducing a transition with a rebuild of every Rust package?
> 
> A security bug in the standard library would require rebuilding and
> shipping the universe, so yeah I'm pretty sure it's quite fundamental.

it would also be pretty much untenable for unstable/testing:
- rustc releases every 6 weeks
- rustc release N requires N or N-1 to build
- we frequently need to multiple rustc uploads for a single version to
  iron out arch-related issues (some of which only show up when building
  particular crates)
- we have > 2k source package in the rust- namespace alone

keeping rustc somewhat current is already a big effort..


signature.asc
Description: PGP signature

Re: Understanding what's missing for Rust dynamic linking (was: Proposal for how to deal with Go/Rust/etc security bugs)

[Luca Boccassi]

On Thu, 25 Jan 2024 at 18:22, Gard Spreemann  wrote:
>
> Hello.
>
> Paul Wise  writes:
>
> > On Thu, 2024-01-25 at 00:24 +, Wookey wrote:
> >
> >> People keep telling us (@ARM) how marvellous Rust is, and we keep
> >> telling them that it's useless in the real world until it sorts out
> >> the stable ABI/dynamic linking problem.
> >
> > IIRC that has been worked on for some years now, and IIRC
> > the static linking wiki page has some references about this.
> >
> > https://wiki.debian.org/StaticLinking
>
> This reminded me that I'm not even sure that I fully understand what
> Rust's remaining technical obstacles to achieving dynamic linking (at
> least within Debian) are. I'm ignoring the potential cultural or
> political issues that have been alluded to by others. My understanding –
> and please do correct me! – has been that three components are missing:
>
> (1) A stable ABI.
<...>
> From Debian's perspective, is really (1) all that important given that a
> stable release only has to deal with a specific version of the compiler?
> Could we not live with every new version of *just* rustc in sid
> introducing a transition with a rebuild of every Rust package?

A security bug in the standard library would require rebuilding and
shipping the universe, so yeah I'm pretty sure it's quite fundamental.

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Luca Boccassi]

On Thu, 25 Jan 2024 at 16:11, Russ Allbery  wrote:
>
> Simon Josefsson  writes:
>
> > I want to explore if there is a possibility to change status quo, and
> > what would be required to do so.
>
> > Given how often gnulib is vendored for C code in Debian, and other
> > similar examples, I don't think of this problem as purely a Go/Rust
> > problem.  The parallel argument that we should not support coreutils,
> > sed, tar, gzip etc because they included vendored copies of gnulib code
> > is not reasonable.
>
> Since there are now a bunch of messages on this thread of people grumbling
> about Rust and Go and semi-proposing not even trying to package that
> software (and presumably removing python3-cryptography and everything that
> depends on it? I'm not sure where people think this argument is going), I
> wanted to counterbalance that by saying I completely agree with Simon's
> exploration here.
>
> Rebuilding a bunch of software after a security fix is not a completely
> intractable problem that we have no idea how to even approach.  It's just
> CPU cycles and good metadata plus ensuring that our software can be
> rebuilt, something that we already promise.  Some aspects of making this
> work will doubtless be *annoying*, but it doesn't seem outside of our
> capabilities as a project.

Well, CPu cycles and metadata are not quite the whole story though. It
also takes a huge amount of extra engineering effort, both one-time to
completely rearchitect and reimplement the build and release
infrastructures to deal with this at scale, and ongoing to maintain
the additional complexities for the rest of the project's life. As you
well know, developers time is by far the scarcest resource of them
all.
Also, it means downloading and installing GBs of packages on every
security update, instead of a few KBs, for every user. While not as
bad as the resource issue, it's still not great.

Finally, as an opinion, quite frankly, it's not a very good design.
You can already run a distribution that does static linking for
everything and requires rebuilding the whole universe and then some
every time there's a one character change anywhere - take Yocto and
configure it to use Musl. Having used it for a long time now, it's
_really_ not good and I wouldn't recommend it to anybody. Matter of
taste of course.

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Russ Allbery]

Simon Josefsson  writes:

> I want to explore if there is a possibility to change status quo, and
> what would be required to do so.

> Given how often gnulib is vendored for C code in Debian, and other
> similar examples, I don't think of this problem as purely a Go/Rust
> problem.  The parallel argument that we should not support coreutils,
> sed, tar, gzip etc because they included vendored copies of gnulib code
> is not reasonable.

Since there are now a bunch of messages on this thread of people grumbling
about Rust and Go and semi-proposing not even trying to package that
software (and presumably removing python3-cryptography and everything that
depends on it? I'm not sure where people think this argument is going), I
wanted to counterbalance that by saying I completely agree with Simon's
exploration here.

Rebuilding a bunch of software after a security fix is not a completely
intractable problem that we have no idea how to even approach.  It's just
CPU cycles and good metadata plus ensuring that our software can be
rebuilt, something that we already promise.  Some aspects of making this
work will doubtless be *annoying*, but it doesn't seem outside of our
capabilities as a project.

Dealing with older versions is of course much more of a problem,
particularly if upstream is not backporting security fixes, but this is a
problem is inherent in having stable releases, that upstreams have been
grumbly about long before either Rust or Go even existed, and that we have
nonetheless dealt with throughout the whole history of Debian.  There is
no one-size-fits-all solution, but we have historically managed to muddle
through in a mostly acceptable way.

-- 
Russ Allbery (r...@debian.org)  

Understanding what's missing for Rust dynamic linking (was: Proposal for how to deal with Go/Rust/etc security bugs)

[Gard Spreemann]

Hello.

Paul Wise  writes:

> On Thu, 2024-01-25 at 00:24 +, Wookey wrote:
>
>> People keep telling us (@ARM) how marvellous Rust is, and we keep
>> telling them that it's useless in the real world until it sorts out
>> the stable ABI/dynamic linking problem.
>
> IIRC that has been worked on for some years now, and IIRC
> the static linking wiki page has some references about this.
>
> https://wiki.debian.org/StaticLinking

This reminded me that I'm not even sure that I fully understand what
Rust's remaining technical obstacles to achieving dynamic linking (at
least within Debian) are. I'm ignoring the potential cultural or
political issues that have been alluded to by others. My understanding –
and please do correct me! – has been that three components are missing:

(1) A stable ABI.

(2) A way of dealing with generic types/functions across dynamic linking
boundaries.

(3) A way of dealing with macros across dynamic linking boundaries.

From Debian's perspective, is really (1) all that important given that a
stable release only has to deal with a specific version of the compiler?
Could we not live with every new version of *just* rustc in sid
introducing a transition with a rebuild of every Rust package?

As for (2): Since Debian has the privilege of having a complete overview
of the "closed system" of all Rust packages that we need to consider,
could we not conceivably make a pass across all Rust packages in the
entire archive and record every concrete version of every generic
type/function ever used? Then when a particular library package is
built, we would force the monomorphization of all the relevant
types/functions in that shared library's public interface. This would
require tooling support from upstream to force generation of
monomorphized versions of types/functions when building each shared
library, but that in itself does not seem impossible. (As a curious
effect, the introduction of a new package may then trigger the need to
rebuild some of its own dependencies due to new monomorphic versions of
functions being needed that had not been seen in the archive before.)

Similarly, for (3), could we expand every macro in every library as
needed by every depending package in the archive?


Just trying to get a handle on how far we are from a solution of sorts,
apologies for any stupid questions.


 Best,
 Gard



signature.asc
Description: PGP signature

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Marco d'Itri]

On Jan 25, Wookey  wrote:

> Luca is quite right here. Ultimately this can only be fixed by these
> ecosystems understanding that software in these languages cannot be
> sensibly used in distributions until they support modularity and
> stability. The rust people make the excuse that they are 'too new' to
> define a stable ABI. That was fair enough for a while, but it's
> getting to be quite a thin excuse at this point. I think the real
The problem here is that many of these upstream developers actually see 
this as a feature: they are happy to not have "old versions" of their 
software shipped by distributions, because this way they can tell users 
to just download the latest pre-built binaries from github or an 
appimage and not care about supporting older releases.

And while this somehow often works for stand-alone desktop or web
applications, it is hell for daemons or other system services which need 
integration with the OS.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Paul Wise]

On Thu, 2024-01-25 at 00:24 +, Wookey wrote:

> People keep telling us (@ARM) how marvellous Rust is, and we keep
> telling them that it's useless in the real world until it sorts out
> the stable ABI/dynamic linking problem.

IIRC that has been worked on for some years now, and IIRC
the static linking wiki page has some references about this.

https://wiki.debian.org/StaticLinking

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Wookey]

On 2024-01-24 13:29 +0100, Simon Josefsson wrote:
> Luca Boccassi  writes:
> 
> > There's always option B: recognize that the Rust/Go ecosystems are not
> > designed to be compatible with the Linux distributions model

Luca is quite right here. Ultimately this can only be fixed by these
ecosystems understanding that software in these languages cannot be
sensibly used in distributions until they support modularity and
stability. The rust people make the excuse that they are 'too new' to
define a stable ABI. That was fair enough for a while, but it's
getting to be quite a thin excuse at this point. I think the real
issue now is that the people doing the work like their 'very
convenient for developers, who cares about anyone else' model so
no-one in that community is very bothered to fix it. People like us
probably need to put in time to help them.

> Go seems to have supported shared libraries since around ca 2015:
> 
> https://go.dev/talks/2015/state-of-go-may.slide#13
> https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit

OK. So at least they _have_ a shared library mechanism we could
use. Does anyone know what is stopping debian from using it? Can we
just start requiring go stuff to build shared libraries in policy?
I've done this before with C libraries where upstream only ever
shipped a static library so the makefiles had to be expanded a bit,
but it's not a lot of work. Is that all we need in Go, or is there
some larger issue or much more work needed?

> > There are many ways to ship applications today that are much better
> > suited for these models, like Flatpak, where the maintenance burden is
> > shifted onto those who choose to opt in to such ecosystems.

Simply ignoring them until they get a clue would be nice, but I don't
think it's very practical at this point, and would be a disservice to
our users.

But I am certainly in favour of directing effort to try to get
relevant change upstream in the ecosystems. i.e stop chopping long
enough to sharpen our axe. I have done some of this in the Rust
ecosystem, and I know there is quite a lot of corporate pressure being
applied. People keep telling us (@ARM) how marvellous Rust is, and we keep
telling them that it's useless in the real world until it sorts out
the stable ABI/dynamic linking problem.

I guess it might be time for another go to see if we can get some traction.

Wookey
-- 
Principal hats:  Debian, Wookware, ARM
http://wookware.org/


signature.asc
Description: PGP signature

Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)

[Marco d'Itri]

On Jan 24, Peter Pentchev  wrote:

> This might be a minority, optimistic, rose-tinted-glasses kind of
> opinion, but I believe that the state of the Rust ecosystem today
> (I have no experience with the Go one) is quite similar to what Perl and
> Python modules were 25, 20, bah, even 15 years ago. Gradually, with time,
I am not familiar with the Python ecosystem, but I have been writing 
Perl and packaging software with Perl dependencies for over 25 years and 
I can confidently say that this is not true.
Perl libraries ("modules") generally never had the API instability that 
I have seen in Rust libraries (but much less in Go, I believe).
In my experience forward compatibility has always been very important in 
the Perl ecosystem.

BTW, for the past couple of years I have been presenting to my other 
community, the network operators, about some of my Debian work and these 
problems with integrating complex Rust software in distributions, e.g.
https://www.linux.it/~md/text/rpki-validators-euroix2023.pdf .

-- 
ciao,
Marco


signature.asc
Description: PGP signature

static linking, modularity, and community (was: Proposal for how to deal with Go/Rust/etc security bugs)

[G. Branden Robinson]

[follow-ups should probably go to -project, but I'm not setting my
headers to try to force that]

At 2024-01-24T16:57:06+0100, Simon Josefsson wrote:
> One could equally well make the argument that distributors should care
> about the Go/Rust ecosystems, and make whatever changes needed in
> order to support them.  Those changes are what I'm trying to explore
> here.

Exploration is one thing; advocacy is another.  Please be clear which
role you're taking in any given discussion.  (If you switch hats
midstream, be explicit about it).

[rearranging]
> Speaking as a C person (I know little about Go/Rust), getting stable
> ABIs, dynamic linking [...] right is not simple, and we've been
> working on that for 20+ years consuming plenty of human resources on
> the way.

That's true, but think about the goal that stable ABIs and dynamic
shared objects achieve: software modularity in the field at runtime.  I
trust that I don't have to argue the benefits of modular design to any
even slightly experienced software developer; its benefits have been
understood for about fifty years.

(Just to get this on the record, and for obligatory technical content:
I've studied Go a little and Rust a bit more.  I think that, _as
languages_, they're likely both superior to C in most respects.  [If you
want to visualize PDP-11 assembler output and stack layout as you code,
both are likely ill-suited to the purpose, as standard C itself
increasingly is.]  In any case they are well worth a software
professional's time to look at, in my opinion.  That doesn't mean you
have to endorse their adoption.)

> and security upgrades

And there's the payoff of dynamic modularity, if you will.

If the origin of a statically linked object, whatever its language of
implementation, is not on the ball _for any reason_, the exposure of a
security vulnerability in it or anything upon which it depends expands a
temporal window of attack on affected systems.  We _know_ that certain
actors in the global IT sector hoard undisclosed vulnerabilities for
deployment later, at opportune times for extortion or data exfiltration
against targets of interest.

Next, consider some of the reasons why the origin of a statically linked
object might come "off the ball".  Maybe they got fired or laid off.
Maybe they burned out.  Maybe they died.  Maybe they've been told the
project is supposed to become part of their "20% time"[sic].  Maybe it's
just "not a priority" to their employer.  Maybe they had a baby or
finally went on that long-planned vacation to a tropical island.

If you've worked in Big IT you know that you can predict that when your
business unit gets a new director, _you_ get a new direction.  Directors
and other senior managers justify their huge salaries and benefit
packages by being "impactful", and one reliable way to be "impactful" is
by handing down dramatic decrees.  (All of the top capitalists are
gamblers; one can easily perceive this in their personalities and
rhetoric.)  At one place I worked such a thing happened twice in less
than a year, I think.  We were going to "drop all Linux" and rebase
everything on FreeBSD.  Then that director departed the company and
FreeBSD was out the window--back to Linux.[1]

Worse, when any of these disruptions happens to the origin of a module
that employs static linking, it can take an indefinite amount of time to
_find out_ that such has even taken place.  So its user community may
lose time while it politely waits for their upstream to check in from
vacation or whatever, because they don't want to be rude, or seen as
hijacking the project, or similar.  And even that assumes that the right
people in the community have sufficient expertise to reasonably
approximate the usual construction and release procedures, which can
vary not just in the technology of the implementation system, but in the
hosting/development site.

Consider the motivations and pressures that professionalized, corporate
software is developed under.  Any cost of development that can be
externalized experiences an unrelenting force in that direction.

  An economic profit is the difference between the revenue received from
  sales and the explicit costs of producing its goods and services, as
  well as any opportunity costs.[2]

Opportunity costs are an interesting subject.[3]  They are often
difficult or even impossible to measure.  In business culture, they are
often therefore utterly neglected, even when they are known to exist.
This is why firms pollute, know they pollute, deny to the public that
they pollute, argue with everyone who challenges them about the
consequences of their pollution, and ultimately attempt to escape
liability for their subjection of others--ultimately, human beings and
the environment generally--to their negative externalities.  The
emphasis on profit and a desire by the wealthiest in society to preserve
the prevailing economic system is also why very concept of negative
externalities is under-emphasized or 

Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)

[Peter Pentchev]

On Wed, Jan 24, 2024 at 01:01:34PM +, Luca Boccassi wrote:
> On Wed, 24 Jan 2024 at 12:26, Johannes Schauer Marin Rodrigues
>  wrote:
> >
> > Hi,
> >
> > Quoting Luca Boccassi (2024-01-24 12:59:38)
> > > There's always option B: recognize that the Rust/Go ecosystems are not
> > > designed to be compatible with the Linux distributions model, and are 
> > > instead
> > > designed to be as convenient as possible for a _single_ application 
> > > developer
> > > and its users - at the detriment of everybody else - and for large
> > > corporations that ship a handful of applications with swathes of engineers
> > > that can manage the churn, and it is not made nor intended to scale to 
> > > ~60k
> > > packages or whatever number we have today in unstable. And simply stop
> > > attempting to merge these two incompatible ecosystems against their will, 
> > > at
> > > the very least until and unless they reach feature and stability parity 
> > > with
> > > the C/C++ ecosystems - stable API/ABI and dynamic linking by default.
> > >
> > > There are many ways to ship applications today that are much better
> > > suited for these models, like Flatpak, where the maintenance burden is
> > > shifted onto those who choose to opt in to such ecosystems.
> >
> > how does that work for those applications that require rust, go and friends?
> > Are you proposing that everything that needs them should be be distributed 
> > by a
> > flatpak or similar mechanism instead?
> 
> Those applications are not shipped in the distribution. If somebody
> wants to use them, they'll have to figure it out, just like for
> everything else that is not shipped in the distribution, which is
> already a subset of all available software in the world.
> 
> > Just a few days ago I tried building mesa from experimental (otherwise there
> > are severe graphics glitches on my platform) on bookworm and everything 
> > worked
> > except its rust build dependencies.  I had no luck trying to backport those
> > parts of rust that I needed and even if it had worked, it would've meant
> > backporting dozens of rust packages just to have a backport of mesa. It 
> > seemed
> > way simpler to just disable the mesa component that requires rust and call 
> > it a
> > day. But that probably doesn't work for all applications and maybe sometimes
> > the component written in rust is actually what you want. And in case of 
> > mesa, a
> > flatpak of it probably would also not've helped as that would've meant that 
> > I
> > need to run everything that I want to run with a more modern mesa based on 
> > that
> > flatpak, right?
> >
> > So how is option B a solution in practice?
> 
> You have found first-hand the exact problem with this ecosystem. Now
> try to imagine doing that for 30k packages, all vendoring, pinning and
> static linking against each other at different, incompatible versions.
> Again, the solution in practice is to simply disable or patch out
> those components, and if somebody needs them, they can complain to
> upstream and ask them for a solution, if there is one. If there isn't,
> though luck. And yes, that is not great - but neither are all other
> options, so it seems much wiser to me to push responsibility where it
> belongs - with the upstreams choosing to use such ecosystems, and the
> owner of said ecosystems choosing to make them incompatible with the
> distribution model, as they are in the best position to solve the
> problem(s) that happen due to their design choices.

This might be a minority, optimistic, rose-tinted-glasses kind of
opinion, but I believe that the state of the Rust ecosystem today
(I have no experience with the Go one) is quite similar to what Perl and
Python modules were 25, 20, bah, even 15 years ago. Gradually, with time,
the module authors realized that if they wanted people to use their
modules, they would eventually have to settle on a stable API and
only make incompatible changes very rarely. With time, with this
realization trickling up and down across the most used libraries,
things slowly start to get better.

Yes, even now there are what can only be called "transitions" in
the Perl and Python Debian packaging - some often-used module
makes an incompatible change and the packagers need to wait until
all the other packaged modules have either caught up or it has somehow
been proven through testing that some of the dependent modules are
not really affected by the incompatible change. In my experience,
nowadays this happens much, much more rarely than 10 or 15 years ago.

I have no idea how long it will take the Rust ecosystem to realize
that. I know that some of the most widely used modules have already
done that, some of them have been at the same 0.x.* or even 1.*
version for years (yes, really, years). So... here's hoping.

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.net r...@debian.org p...@storpool.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C 

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Simon Josefsson]

Luca Boccassi  writes:

> On Wed, 24 Jan 2024 at 13:34, Simon Josefsson  wrote:
>>
>> Luca Boccassi  writes:
>>
>> >> Having reflected a bit, and learned through my own experience and
>> >> others' insights [1] that Go Build-Depends are not transitive, I'd like
>> >> to update my proposal on how to handle a security bug in any Go/Rust/etc
>> >> package and the resulting package rebuilds:
>> >
>> > There's always option B: recognize that the Rust/Go ecosystems are not
>> > designed to be compatible with the Linux distributions model
>>
>> Definitely - that's roughly the model we have today, right?  So no
>> action needed to preserve status quo of option B.
>>
>> I want to explore if there is a possibility to change status quo, and
>> what would be required to do so.
>
> What's required is talking to the language ecosystem owners and
> convince them to support a stable ABI and dynamic linking, and in
> general to care about the distribution use case. Otherwise it's just
> an unwinnable uphill battle that consumes a ton of scarce resources
> (developers time), and is simply hopeless.

One could equally well make the argument that distributors should care
about the Go/Rust ecosystems, and make whatever changes needed in order
to support them.  Those changes are what I'm trying to explore here.

Speaking as a C person (I know little about Go/Rust), getting stable
ABIs, dynamic linking and security upgrades right is not simple, and
we've been working on that for 20+ years consuming plenty of human
resources on the way.

/Simon


signature.asc
Description: PGP signature

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Luca Boccassi]

On Wed, 24 Jan 2024 at 13:34, Simon Josefsson  wrote:
>
> Luca Boccassi  writes:
>
> >> Having reflected a bit, and learned through my own experience and
> >> others' insights [1] that Go Build-Depends are not transitive, I'd like
> >> to update my proposal on how to handle a security bug in any Go/Rust/etc
> >> package and the resulting package rebuilds:
> >
> > There's always option B: recognize that the Rust/Go ecosystems are not
> > designed to be compatible with the Linux distributions model
>
> Definitely - that's roughly the model we have today, right?  So no
> action needed to preserve status quo of option B.
>
> I want to explore if there is a possibility to change status quo, and
> what would be required to do so.

What's required is talking to the language ecosystem owners and
convince them to support a stable ABI and dynamic linking, and in
general to care about the distribution use case. Otherwise it's just
an unwinnable uphill battle that consumes a ton of scarce resources
(developers time), and is simply hopeless.

> Given how often gnulib is vendored for C code in Debian, and other
> similar examples, I don't think of this problem as purely a Go/Rust
> problem.  The parallel argument that we should not support coreutils,
> sed, tar, gzip etc because they included vendored copies of gnulib code
> is not reasonable.

As it was already explained, this is a misleading example that has
nothing to do with the problems of the Go/Rust ecosystem.

Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)

[Jeremy Stanley]

On 2024-01-24 13:26:49 +0100 (+0100), Johannes Schauer Marin Rodrigues wrote:
[...]
> how does that work for those applications that require rust, go
> and friends? Are you proposing that everything that needs them
> should be be distributed by a flatpak or similar mechanism
> instead?
> 
> Just a few days ago I tried building mesa from experimental
> (otherwise there are severe graphics glitches on my platform) on
> bookworm and everything worked except its rust build dependencies.
> I had no luck trying to backport those parts of rust that I needed
> and even if it had worked, it would've meant backporting dozens of
> rust packages just to have a backport of mesa.
[...]

Another practical example is Python applications/libs increasingly
integrating performance or security critical routines (re)written in
Rust. The python-cryptography source package in sid build-depends on
cargo and a dozen Rust libs. I didn't bother to pull a rdeps list
for it, but can guarantee a vast swath of the python3-* packages in
Debian depend on python3-cryptography, and this particular problem
will only get worse as the CPython core devs see the rising
popularity of the externally-developed cryptography library as a
good reason to strip any remnants of cryptographic modules and
bindings from the stdlib.
-- 
Jeremy Stanley


signature.asc
Description: PGP signature

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Gioele Barabucci]

On 24/01/24 14:01, Luca Boccassi wrote:

how does that work for those applications that require rust, go and friends?
Are you proposing that everything that needs them should be be distributed by a
flatpak or similar mechanism instead?


Those applications are not shipped in the distribution. If somebody
wants to use them, they'll have to figure it out, just like for
everything else that is not shipped in the distribution, which is
already a subset of all available software in the world.


The risk is that that subset will get even smaller and miss out on new 
developments: Debian will then be stuck as a "C distribution" and with C 
& Co becoming more and more legacy, that means that Debian will become a 
"legacy distribution".


It is nice to know that, even if the paradigms are changing, one can 
still apt install caddy, etcd, acmetool, influxdb, prometheus, sq, 
firefox...


--
Gioele Barabucci

Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)

[Luca Boccassi]

On Wed, 24 Jan 2024 at 12:26, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi,
>
> Quoting Luca Boccassi (2024-01-24 12:59:38)
> > There's always option B: recognize that the Rust/Go ecosystems are not
> > designed to be compatible with the Linux distributions model, and are 
> > instead
> > designed to be as convenient as possible for a _single_ application 
> > developer
> > and its users - at the detriment of everybody else - and for large
> > corporations that ship a handful of applications with swathes of engineers
> > that can manage the churn, and it is not made nor intended to scale to ~60k
> > packages or whatever number we have today in unstable. And simply stop
> > attempting to merge these two incompatible ecosystems against their will, at
> > the very least until and unless they reach feature and stability parity with
> > the C/C++ ecosystems - stable API/ABI and dynamic linking by default.
> >
> > There are many ways to ship applications today that are much better
> > suited for these models, like Flatpak, where the maintenance burden is
> > shifted onto those who choose to opt in to such ecosystems.
>
> how does that work for those applications that require rust, go and friends?
> Are you proposing that everything that needs them should be be distributed by 
> a
> flatpak or similar mechanism instead?

Those applications are not shipped in the distribution. If somebody
wants to use them, they'll have to figure it out, just like for
everything else that is not shipped in the distribution, which is
already a subset of all available software in the world.

> Just a few days ago I tried building mesa from experimental (otherwise there
> are severe graphics glitches on my platform) on bookworm and everything worked
> except its rust build dependencies.  I had no luck trying to backport those
> parts of rust that I needed and even if it had worked, it would've meant
> backporting dozens of rust packages just to have a backport of mesa. It seemed
> way simpler to just disable the mesa component that requires rust and call it 
> a
> day. But that probably doesn't work for all applications and maybe sometimes
> the component written in rust is actually what you want. And in case of mesa, 
> a
> flatpak of it probably would also not've helped as that would've meant that I
> need to run everything that I want to run with a more modern mesa based on 
> that
> flatpak, right?
>
> So how is option B a solution in practice?

You have found first-hand the exact problem with this ecosystem. Now
try to imagine doing that for 30k packages, all vendoring, pinning and
static linking against each other at different, incompatible versions.
Again, the solution in practice is to simply disable or patch out
those components, and if somebody needs them, they can complain to
upstream and ask them for a solution, if there is one. If there isn't,
though luck. And yes, that is not great - but neither are all other
options, so it seems much wiser to me to push responsibility where it
belongs - with the upstreams choosing to use such ecosystems, and the
owner of said ecosystems choosing to make them incompatible with the
distribution model, as they are in the best position to solve the
problem(s) that happen due to their design choices.

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Praveen Arimbrathodiyil]

On 24/01/24 5:59 pm, Simon Josefsson wrote:

Does anyone know of a shared library in a Debian package written in Go?
I've only encountered the vendored approach to ship Go libraries.


libgovarnam1 is shipped as a shared library
https://packages.debian.org/sid/amd64/libgovarnam1/filelist



OpenPGP_0x8F53E0193B294B75.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Simon Josefsson]

Luca Boccassi  writes:

>> Having reflected a bit, and learned through my own experience and
>> others' insights [1] that Go Build-Depends are not transitive, I'd like
>> to update my proposal on how to handle a security bug in any Go/Rust/etc
>> package and the resulting package rebuilds:
>
> There's always option B: recognize that the Rust/Go ecosystems are not
> designed to be compatible with the Linux distributions model

Definitely - that's roughly the model we have today, right?  So no
action needed to preserve status quo of option B.

I want to explore if there is a possibility to change status quo, and
what would be required to do so.

Given how often gnulib is vendored for C code in Debian, and other
similar examples, I don't think of this problem as purely a Go/Rust
problem.  The parallel argument that we should not support coreutils,
sed, tar, gzip etc because they included vendored copies of gnulib code
is not reasonable.

> and are instead designed to be as convenient as possible for a
> _single_ application developer and its users - at the detriment of
> everybody else - and for large corporations that ship a handful of
> applications with swathes of engineers that can manage the churn, and
> it is not made nor intended to scale to ~60k packages or whatever
> number we have today in unstable. And simply stop attempting to merge
> these two incompatible ecosystems against their will, at the very
> least until and unless they reach feature and stability parity with
> the C/C++ ecosystems - stable API/ABI and dynamic linking by default.

Go seems to have supported shared libraries since around ca 2015:

https://go.dev/talks/2015/state-of-go-may.slide#13
https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit

Does anyone know of a shared library in a Debian package written in Go?
I've only encountered the vendored approach to ship Go libraries.

> There are many ways to ship applications today that are much better
> suited for these models, like Flatpak, where the maintenance burden is
> shifted onto those who choose to opt in to such ecosystems.

Or simply 'go install ...', it works remarkably well.

However, this is orthogonal to how we support the Go code that is in
Debian already.

/Simon


signature.asc
Description: PGP signature

Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-01-24 12:59:38)
> There's always option B: recognize that the Rust/Go ecosystems are not
> designed to be compatible with the Linux distributions model, and are instead
> designed to be as convenient as possible for a _single_ application developer
> and its users - at the detriment of everybody else - and for large
> corporations that ship a handful of applications with swathes of engineers
> that can manage the churn, and it is not made nor intended to scale to ~60k
> packages or whatever number we have today in unstable. And simply stop
> attempting to merge these two incompatible ecosystems against their will, at
> the very least until and unless they reach feature and stability parity with
> the C/C++ ecosystems - stable API/ABI and dynamic linking by default.
> 
> There are many ways to ship applications today that are much better
> suited for these models, like Flatpak, where the maintenance burden is
> shifted onto those who choose to opt in to such ecosystems.

how does that work for those applications that require rust, go and friends?
Are you proposing that everything that needs them should be be distributed by a
flatpak or similar mechanism instead?

Just a few days ago I tried building mesa from experimental (otherwise there
are severe graphics glitches on my platform) on bookworm and everything worked
except its rust build dependencies.  I had no luck trying to backport those
parts of rust that I needed and even if it had worked, it would've meant
backporting dozens of rust packages just to have a backport of mesa. It seemed
way simpler to just disable the mesa component that requires rust and call it a
day. But that probably doesn't work for all applications and maybe sometimes
the component written in rust is actually what you want. And in case of mesa, a
flatpak of it probably would also not've helped as that would've meant that I
need to run everything that I want to run with a more modern mesa based on that
flatpak, right?

So how is option B a solution in practice?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)

[Luca Boccassi]

On Wed, 24 Jan 2024 at 11:42, Simon Josefsson  wrote:
>
> Simon Josefsson  writes:
>
> >> > My naive approach on how to fix a security problem in package X
> >> > which is
> >> > statically embedded into other packages A, B, C, ... would be to
> >> > rebuild
> >> > the transitive closure of all packages that Build-Depends on X and
> >> > publish a security update for all those packages.
> ...
> > I realized that there is one problem with my approach: consider if
> > package A was built via Build-Depends package B of version X and that
> > later package B is upgraded to X+1 in Debian.  Then if a security
> > problem happens in B we need to rebuild A. It may be that package A no
> > longer builds due to some incompatibility between version X and X+1 of
> > B.  This would not be noticed until a full rebuild of an archive is
> > done, or when the security teams wants to rebuild the transitive
> > closure of the Build-Depends graph for a package.
>
> Having reflected a bit, and learned through my own experience and
> others' insights [1] that Go Build-Depends are not transitive, I'd like
> to update my proposal on how to handle a security bug in any Go/Rust/etc
> package and the resulting package rebuilds:

There's always option B: recognize that the Rust/Go ecosystems are not
designed to be compatible with the Linux distributions model, and are
instead designed to be as convenient as possible for a _single_
application developer and its users - at the detriment of everybody
else - and for large corporations that ship a handful of applications
with swathes of engineers that can manage the churn, and it is not
made nor intended to scale to ~60k packages or whatever number we have
today in unstable. And simply stop attempting to merge these two
incompatible ecosystems against their will, at the very least until
and unless they reach feature and stability parity with the C/C++
ecosystems - stable API/ABI and dynamic linking by default.

There are many ways to ship applications today that are much better
suited for these models, like Flatpak, where the maintenance burden is
shifted onto those who choose to opt in to such ecosystems.

Re: Proposal for how to deal with Go/Rust/etc security bugs

[Praveen Arimbrathodiyil]

On 24/01/24 2:07 pm, Simon Josefsson wrote:

Yes, for a low-level Go package (e.g., golang-golang-x-net-dev), this
will mean rebuilding almost all of the Go packages in Debian and publish
them in a security advisory.

This algorithm can be optimized (i.e., reduce the number of packages to
publish in an advisory) by either of:

 1) using information from Built-Using: (which was not designed for
this purpose, so this is fragile) or *.buildinfo.

 2) by dropping all 'Architecture: all' packages that does not embedd
the buggy code.

The last optimization 2) would reduce the number of Go packages to
publish significantly, as it would drop most golang-*-dev packages.  I
think this actually makes this process feasible in practice, as there
are relatively few binary packages written in Go.


I was also wondering about this, the actual number of arch:any go 
packages is much smaller if we skip arch:all *-dev packages so this 
should be a smaller number of rebuilds than what is currently 
considered. We can make this even smaller by choosing a limited number 
of packages for security support, for example caddy, soh etc, which 
would already be better than not providing any security update at all.


OpenPGP_0x8F53E0193B294B75.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Proposal for how to deal with Go/Rust/etc security bugs (was: Re: Limited security support for Go/Rust? Re ssh3)

[Simon Josefsson]

Simon Josefsson  writes:

>> > My naive approach on how to fix a security problem in package X
>> > which is
>> > statically embedded into other packages A, B, C, ... would be to
>> > rebuild
>> > the transitive closure of all packages that Build-Depends on X and
>> > publish a security update for all those packages.
...
> I realized that there is one problem with my approach: consider if
> package A was built via Build-Depends package B of version X and that
> later package B is upgraded to X+1 in Debian.  Then if a security
> problem happens in B we need to rebuild A. It may be that package A no
> longer builds due to some incompatibility between version X and X+1 of
> B.  This would not be noticed until a full rebuild of an archive is
> done, or when the security teams wants to rebuild the transitive
> closure of the Build-Depends graph for a package.

Having reflected a bit, and learned through my own experience and
others' insights [1] that Go Build-Depends are not transitive, I'd like
to update my proposal on how to handle a security bug in any Go/Rust/etc
package and the resulting package rebuilds:

  To fix a security problem in package X, which is used during build
  (through statical linking, vendoring, or some other mechanism) to
  build package A, B, C, ... you need to rebuild all packages that
  Build-Depends on X (lets call this step 1) and all packages that
  Build-Depends on any of the packages that will be rebuilt during step
  1.  This rebuild process is iterated until no more packages remains to
  be rebuild, and all packages have been rebuilt using newly rebuild
  packages.  If there are cyclical dependencies (which unfortunately are
  common), you have to loop until all packages have been rebuilt with a
  clean dependency chain, and detect the loop and stop rebuilding.  If
  there are FTBFS errors during the rebuilds, this will have to be
  patched too.

Yes, for a low-level Go package (e.g., golang-golang-x-net-dev), this
will mean rebuilding almost all of the Go packages in Debian and publish
them in a security advisory.

This algorithm can be optimized (i.e., reduce the number of packages to
publish in an advisory) by either of:

1) using information from Built-Using: (which was not designed for
   this purpose, so this is fragile) or *.buildinfo.

2) by dropping all 'Architecture: all' packages that does not embedd
   the buggy code.

The last optimization 2) would reduce the number of Go packages to
publish significantly, as it would drop most golang-*-dev packages.  I
think this actually makes this process feasible in practice, as there
are relatively few binary packages written in Go.

This method applies to non-Go/Rust too, if there are such security
problems and reverse build chains.

I think all of this (except maybe the optimization 2) which requires
code comparisons) can be automated in a GitLab pipeline for higher
confidence of the result.

/Simon

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=806481#58


signature.asc
Description: PGP signature