Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
On Mon, Jul 11, 2016 at 05:41:52PM +1000, Dmitry Smirnov wrote:
> On Monday, 11 July 2016 9:22:12 AM AEST Florian Weimer wrote:
> > Hmm. I poked at a few packages, and here is what I found:
> > golang-siphash-dev does not have any Built-Using header.
> > golang-gopkg-tylerb-graceful.v1-dev does not list golang-x-text,
> > although its dependency golang-golang-x-net-dev was built using it.
> > (I'm looking at unstable.)
>
> But you are looking at wrong packages. -dev packages are just sources that
> strictly speaking are not "built" but more like "validated" on build time.
>
> You do not need to re-build source/-dev packages so they do not have Built-
> Using header intentionally. What you need to be looking at is arch:any binary
> packages built from go sources involving multiple libraries.
>
> Examples of Golang executables include docker.io, containerd, etcd, grafana,
> runc, acbuild, docker2aci, influxdb, rkt, consul, fleet, nomad, skydns,
> kubernetes-{node|master|client}, etc.
To have some hard numbers: If golang itselfs needs an update, how many
packages need a rebuild?
Cheers,
Moritz
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
On Mon, Jul 11, 2016 at 11:02:59PM +1200, Michael Hudson-Doyle wrote: > A Built-Using field must list the corresponding source package for any > such binary package incorporated during the build..." > > Lots of -dev packages _do_ have Built-Using though, we should try to > kill them off... Could you mention correct use of Built-Using in the Go Packaging policy? https://pkg-go.alioth.debian.org/packaging.html Peter ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
On 11 July 2016 at 19:22, Florian Weimer wrote: > * Michael Hudson-Doyle: > >> On 10 July 2016 at 07:39, Florian Weimer wrote: >>> * Dmitry Smirnov: >>> On Friday, 8 July 2016 8:53:20 AM AEST Florian Weimer wrote: > Part of the problem is that we currently lack a decent way to list all > these reverse dependencies. We can get list of all source packages to re-build from reverse build dependencies. Then it should be possible to filter arch:any packages to bin- NMU. Alternatively Built-Using field could be of help. >>> >>> We already discussed why this doesn't work with the present state of >>> the metadata. >> >> Do you mean the "B-U is only direct dependencies" problem? That's >> fixed now. > > Hmm. I poked at a few packages, and here is what I found: > golang-siphash-dev does not have any Built-Using header. > golang-gopkg-tylerb-graceful.v1-dev does not list golang-x-text, > although its dependency golang-golang-x-net-dev was built using it. > (I'm looking at unstable.) As Dmitry says, -dev packages should not have Built-Using headers at all, https://www.debian.org/doc/debian-policy/ch-relationships.html#s-built-using: "Some binary packages incorporate parts of other packages when built but do not have to depend on those packages. Examples include linking with static libraries or incorporating source code from another package during the build. In this case, the source packages of those other packages are a required part of the complete source (the binary package is not reproducible without them). A Built-Using field must list the corresponding source package for any such binary package incorporated during the build..." Lots of -dev packages _do_ have Built-Using though, we should try to kill them off... > Would this be fixable through a mass rebuild? If there are binary packages with out of date Built-Using, yes, rebuilding will fix. >> Was there something else? > > The dependency-generating script does not end up in Built-Using. > I think we discussed including its version at some point. As above, not really the right place for it? Cheers, mwh ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
On Monday, 11 July 2016 9:22:12 AM AEST Florian Weimer wrote:
> Hmm. I poked at a few packages, and here is what I found:
> golang-siphash-dev does not have any Built-Using header.
> golang-gopkg-tylerb-graceful.v1-dev does not list golang-x-text,
> although its dependency golang-golang-x-net-dev was built using it.
> (I'm looking at unstable.)
But you are looking at wrong packages. -dev packages are just sources that
strictly speaking are not "built" but more like "validated" on build time.
You do not need to re-build source/-dev packages so they do not have Built-
Using header intentionally. What you need to be looking at is arch:any binary
packages built from go sources involving multiple libraries.
Examples of Golang executables include docker.io, containerd, etcd, grafana,
runc, acbuild, docker2aci, influxdb, rkt, consul, fleet, nomad, skydns,
kubernetes-{node|master|client}, etc.
--
All the best,
Dmitry Smirnov.
---
To swallow and follow, whether old doctrine or new propaganda, is a
weakness still dominating the human mind.
-- Charlotte P. Gilman
signature.asc
Description: This is a digitally signed message part.
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
* Michael Hudson-Doyle: > On 10 July 2016 at 07:39, Florian Weimer wrote: >> * Dmitry Smirnov: >> >>> On Friday, 8 July 2016 8:53:20 AM AEST Florian Weimer wrote: Part of the problem is that we currently lack a decent way to list all these reverse dependencies. >>> >>> We can get list of all source packages to re-build from reverse build >>> dependencies. Then it should be possible to filter arch:any >>> packages to bin- >>> NMU. >>> >>> Alternatively Built-Using field could be of help. >> >> We already discussed why this doesn't work with the present state of >> the metadata. > > Do you mean the "B-U is only direct dependencies" problem? That's > fixed now. Hmm. I poked at a few packages, and here is what I found: golang-siphash-dev does not have any Built-Using header. golang-gopkg-tylerb-graceful.v1-dev does not list golang-x-text, although its dependency golang-golang-x-net-dev was built using it. (I'm looking at unstable.) Would this be fixable through a mass rebuild? > Was there something else? The dependency-generating script does not end up in Built-Using. I think we discussed including its version at some point. ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
On 10 July 2016 at 07:39, Florian Weimer wrote: > * Dmitry Smirnov: > >> On Friday, 8 July 2016 8:53:20 AM AEST Florian Weimer wrote: >>> Part of the problem is that we currently lack a decent way to list all >>> these reverse dependencies. >> >> We can get list of all source packages to re-build from reverse build >> dependencies. Then it should be possible to filter arch:any packages to bin- >> NMU. >> >> Alternatively Built-Using field could be of help. > > We already discussed why this doesn't work with the present state of > the metadata. Do you mean the "B-U is only direct dependencies" problem? That's fixed now. Was there something else? Cheers, mwh ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
* Michael Hudson-Doyle: > Also, the way my prototype dh-golang change works, a libgolang* > package Provides a value that contains the abi hash and dependencies > depend on the hash value (via dpkg-shlibdeps), so in that case > figuring out how much to rebuild is a case of "build stuff until > britney stops shouting at you about making packages uninstallable" (I > don't know if that's practical for the way you build security updates > though). What goes into the ABI hash? All export data (which, I believe, includes the body of functions which can be inlined across compilation unit boundaries)? > Over releases, no, I think you're right, but I really hope that > security fixes can at least sometimes preserve ABI (the crypto fixes > in Go 1.6.1 would not break ABI, for example). I looked at this for a few updates, and with the exception of potential inlining (for which I don't have a mental model), it seems to happen surprisingly often for Go itself. ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
On 14 April 2016 at 19:16, Michael Stapelberg wrote: > Thanks for the patch, it’s now merged and uploaded. Awesome, thanks for that. > I’d prefer if you could send such patches in a bug report instead of to > mailing lists which I don’t actively read :). Noted! > In fact, I’d say it’s long overdue to make this package team-maintained. Sounds sensible. > The repository is already in > collab-maint, so if you want to make the necessary changes, please just go > ahead. I don't think I can commit to collab-maint (I'm not a DD or even a DM yet). Cheers, mwh > With regards to the original post, I think we have the same issue that the > haskell packaging community has, since they have the same linking model. > I’ve talked to Joachim Breitner (nomeata) about this a couple years ago and > he mentioned they have some tooling which addresses the issue in a > sufficient way. > > I’d suggest to tackle the problem the same way for Go, and maybe share some > tools if applicable. That said, I won’t have time or motivation to do any of > the work required for this, so volunteers are very welcome. > > On Thu, Apr 14, 2016 at 3:08 AM, Michael Hudson-Doyle > wrote: >> >> On 13 April 2016 at 21:05, Michael Hudson-Doyle >> wrote: >> > On 13 April 2016 at 17:07, Tianon Gravi wrote: >> >> On 12 April 2016 at 21:39, Michael Hudson-Doyle >> >> wrote: >> >>> We could do it without 1) and the consequent re-uploading of every go >> >>> library by using dpkg-query --search a lot, which would be slow I >> >>> guess, but maybe could be done as a fallback? >> >> >> >> I still asking dpkg about file/directory package ownership should be >> >> our primary means of generating this field -- the metadata that dpkg >> >> itself tracks about "which package provided >> >> /usr/share/gocode/src/abc/xyz which I just compiled against" will >> >> always be correct (due to the fact that it really is the single proper >> >> source of truth for such information), where some arbitrary metadata >> >> we add not only clutters up the package metadata as has been >> >> discussed, but much more importantly will have a tendency to "drift" >> >> from the truth, which is something that IMO we shouldn't tolerate for >> >> a field whose primary purpose is knowing when it's necessary to >> >> rebuild, especially for security fixes. Even for really large >> >> packages like Docker (to choose an example that I know off the top of >> >> my head is reasonably hefty WRT deps) we're only talking about maybe >> >> ~200 of these queries at the outside end, and only at build-time, and >> >> only once per build, which IMO is in the realm of reasonable to avoid >> >> yet again uploading a minor fix to every package (moving the metadata >> >> over to the binary packages when we still haven't added the existing >> >> source package metadata to all of them yet) with information that will >> >> have a potential for drifting from the truth or for being too limited >> >> (single package providing multiple namespaces after a repo move, for >> >> example). >> > >> > Yes, all that seems fair. Something like this? >> > http://paste.ubuntu.com/15806327/ -- it's pretty terrible perl, but >> > it's actually arguably simpler than what dh_golang does already! >> >> FWIW, I sent a better version of this patch: >> >> http://lists.alioth.debian.org/pipermail/pkg-go-maintainers/Week-of-Mon-20160411/004304.html >> >> Cheers, >> mwh > > > > > -- > Best regards, > Michael ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
Thanks for the patch, it’s now merged and uploaded. I’d prefer if you could send such patches in a bug report instead of to mailing lists which I don’t actively read :). In fact, I’d say it’s long overdue to make this package team-maintained. The repository is already in collab-maint, so if you want to make the necessary changes, please just go ahead. With regards to the original post, I think we have the same issue that the haskell packaging community has, since they have the same linking model. I’ve talked to Joachim Breitner (nomeata) about this a couple years ago and he mentioned they have some tooling which addresses the issue in a sufficient way. I’d suggest to tackle the problem the same way for Go, and maybe share some tools if applicable. That said, I won’t have time or motivation to do any of the work required for this, so volunteers are very welcome. On Thu, Apr 14, 2016 at 3:08 AM, Michael Hudson-Doyle < michael.hud...@canonical.com> wrote: > On 13 April 2016 at 21:05, Michael Hudson-Doyle > wrote: > > On 13 April 2016 at 17:07, Tianon Gravi wrote: > >> On 12 April 2016 at 21:39, Michael Hudson-Doyle > >> wrote: > >>> We could do it without 1) and the consequent re-uploading of every go > >>> library by using dpkg-query --search a lot, which would be slow I > >>> guess, but maybe could be done as a fallback? > >> > >> I still asking dpkg about file/directory package ownership should be > >> our primary means of generating this field -- the metadata that dpkg > >> itself tracks about "which package provided > >> /usr/share/gocode/src/abc/xyz which I just compiled against" will > >> always be correct (due to the fact that it really is the single proper > >> source of truth for such information), where some arbitrary metadata > >> we add not only clutters up the package metadata as has been > >> discussed, but much more importantly will have a tendency to "drift" > >> from the truth, which is something that IMO we shouldn't tolerate for > >> a field whose primary purpose is knowing when it's necessary to > >> rebuild, especially for security fixes. Even for really large > >> packages like Docker (to choose an example that I know off the top of > >> my head is reasonably hefty WRT deps) we're only talking about maybe > >> ~200 of these queries at the outside end, and only at build-time, and > >> only once per build, which IMO is in the realm of reasonable to avoid > >> yet again uploading a minor fix to every package (moving the metadata > >> over to the binary packages when we still haven't added the existing > >> source package metadata to all of them yet) with information that will > >> have a potential for drifting from the truth or for being too limited > >> (single package providing multiple namespaces after a repo move, for > >> example). > > > > Yes, all that seems fair. Something like this? > > http://paste.ubuntu.com/15806327/ -- it's pretty terrible perl, but > > it's actually arguably simpler than what dh_golang does already! > > FWIW, I sent a better version of this patch: > > http://lists.alioth.debian.org/pipermail/pkg-go-maintainers/Week-of-Mon-20160411/004304.html > > Cheers, > mwh > -- Best regards, Michael ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
On 13 April 2016 at 21:05, Michael Hudson-Doyle wrote: > On 13 April 2016 at 17:07, Tianon Gravi wrote: >> On 12 April 2016 at 21:39, Michael Hudson-Doyle >> wrote: >>> We could do it without 1) and the consequent re-uploading of every go >>> library by using dpkg-query --search a lot, which would be slow I >>> guess, but maybe could be done as a fallback? >> >> I still asking dpkg about file/directory package ownership should be >> our primary means of generating this field -- the metadata that dpkg >> itself tracks about "which package provided >> /usr/share/gocode/src/abc/xyz which I just compiled against" will >> always be correct (due to the fact that it really is the single proper >> source of truth for such information), where some arbitrary metadata >> we add not only clutters up the package metadata as has been >> discussed, but much more importantly will have a tendency to "drift" >> from the truth, which is something that IMO we shouldn't tolerate for >> a field whose primary purpose is knowing when it's necessary to >> rebuild, especially for security fixes. Even for really large >> packages like Docker (to choose an example that I know off the top of >> my head is reasonably hefty WRT deps) we're only talking about maybe >> ~200 of these queries at the outside end, and only at build-time, and >> only once per build, which IMO is in the realm of reasonable to avoid >> yet again uploading a minor fix to every package (moving the metadata >> over to the binary packages when we still haven't added the existing >> source package metadata to all of them yet) with information that will >> have a potential for drifting from the truth or for being too limited >> (single package providing multiple namespaces after a repo move, for >> example). > > Yes, all that seems fair. Something like this? > http://paste.ubuntu.com/15806327/ -- it's pretty terrible perl, but > it's actually arguably simpler than what dh_golang does already! FWIW, I sent a better version of this patch: http://lists.alioth.debian.org/pipermail/pkg-go-maintainers/Week-of-Mon-20160411/004304.html Cheers, mwh ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
On 13 April 2016 at 17:07, Tianon Gravi wrote: > On 12 April 2016 at 21:39, Michael Hudson-Doyle > wrote: >> We could do it without 1) and the consequent re-uploading of every go >> library by using dpkg-query --search a lot, which would be slow I >> guess, but maybe could be done as a fallback? > > I still asking dpkg about file/directory package ownership should be > our primary means of generating this field -- the metadata that dpkg > itself tracks about "which package provided > /usr/share/gocode/src/abc/xyz which I just compiled against" will > always be correct (due to the fact that it really is the single proper > source of truth for such information), where some arbitrary metadata > we add not only clutters up the package metadata as has been > discussed, but much more importantly will have a tendency to "drift" > from the truth, which is something that IMO we shouldn't tolerate for > a field whose primary purpose is knowing when it's necessary to > rebuild, especially for security fixes. Even for really large > packages like Docker (to choose an example that I know off the top of > my head is reasonably hefty WRT deps) we're only talking about maybe > ~200 of these queries at the outside end, and only at build-time, and > only once per build, which IMO is in the realm of reasonable to avoid > yet again uploading a minor fix to every package (moving the metadata > over to the binary packages when we still haven't added the existing > source package metadata to all of them yet) with information that will > have a potential for drifting from the truth or for being too limited > (single package providing multiple namespaces after a repo move, for > example). Yes, all that seems fair. Something like this? http://paste.ubuntu.com/15806327/ -- it's pretty terrible perl, but it's actually arguably simpler than what dh_golang does already! Cheers, mwh ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
On 12 April 2016 at 21:39, Michael Hudson-Doyle wrote: > We could do it without 1) and the consequent re-uploading of every go > library by using dpkg-query --search a lot, which would be slow I > guess, but maybe could be done as a fallback? I still asking dpkg about file/directory package ownership should be our primary means of generating this field -- the metadata that dpkg itself tracks about "which package provided /usr/share/gocode/src/abc/xyz which I just compiled against" will always be correct (due to the fact that it really is the single proper source of truth for such information), where some arbitrary metadata we add not only clutters up the package metadata as has been discussed, but much more importantly will have a tendency to "drift" from the truth, which is something that IMO we shouldn't tolerate for a field whose primary purpose is knowing when it's necessary to rebuild, especially for security fixes. Even for really large packages like Docker (to choose an example that I know off the top of my head is reasonably hefty WRT deps) we're only talking about maybe ~200 of these queries at the outside end, and only at build-time, and only once per build, which IMO is in the realm of reasonable to avoid yet again uploading a minor fix to every package (moving the metadata over to the binary packages when we still haven't added the existing source package metadata to all of them yet) with information that will have a potential for drifting from the truth or for being too limited (single package providing multiple namespaces after a repo move, for example). ♥, - Tianon 4096R / B42F 6819 007F 00F8 8E36 4FD4 036A 9C25 BF35 7DD4 ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
* Paul Tagliamonte: > I don't think B-U is the appropriate place for this. This means if we > didn't change anything in dh-golang, we'd need to binNMU the package before > we can decruft the sources that have a newer versions, dak side. > > With an ftp hat on, I think that's not right. Having the entire build > closure in it would be useful, but B-U is also used by dak to keep sources > we still have binaries related to in the archive. > > We could add it as some sort of binary control header, but that's also > annoying. Less annoying, though. Do you agree that keeping this information would be useful? There's also the option of stuffing these bits into the debug packages. Not as easily analyzed, but it's not so great to pollute the Packages file with such obscure information. ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go
I don't think B-U is the appropriate place for this. This means if we didn't change anything in dh-golang, we'd need to binNMU the package before we can decruft the sources that have a newer versions, dak side. With an ftp hat on, I think that's not right. Having the entire build closure in it would be useful, but B-U is also used by dak to keep sources we still have binaries related to in the archive. We could add it as some sort of binary control header, but that's also annoying. Less annoying, though. On Wed, Apr 6, 2016 at 1:33 PM, Florian Weimer wrote: > * Tianon Gravi: > > > On 5 April 2016 at 14:47, Florian Weimer wrote: > >> We currently need these intermediate dependencies to discover all the > >> affected applications. So perhaps dh_golang needs to construct the > >> transitive closure, instead of listing just immediate build > >> dependencies. If we don't want to put this information into the > >> Packages file, maybe we can keep it in the separate debuginfo > >> packages. > > > > It _should_ be possible to adjust dh_golang to use "go list" in order > > to determine the exact full set of Go packages which the application > > code depends on, and then use _that_ list to cross-reference the files > > in /usr/share/gocode to get the real list of packages for Built-Using > > ( haven't verified whether it's feasible for dh_golang to do this, but > > it's pretty similar to how it's currently using "go list" to gather > > the list of packages to actually build). > > Please also add the version of the dh-golang package, so that we know > what to rebuild if there's a bug in the Built-Using generation. > > ___ > Pkg-go-maintainers mailing list > Pkg-go-maintainers@lists.alioth.debian.org > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers > -- :wq ___ Pkg-go-maintainers mailing list Pkg-go-maintainers@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
