Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go

[Peter Colberg]

On Mon, Jul 11, 2016 at 11:02:59PM +1200, Michael Hudson-Doyle wrote:
> A Built-Using field must list the corresponding source package for any
> such binary package incorporated during the build..."
> 
> Lots of -dev packages _do_ have Built-Using though, we should try to
> kill them off...

Could you mention correct use of Built-Using in the Go Packaging policy?

https://pkg-go.alioth.debian.org/packaging.html

Peter

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go

[Dmitry Smirnov]

On Monday, 11 July 2016 9:22:12 AM AEST Florian Weimer wrote:
> Hmm.  I poked at a few packages, and here is what I found:
> golang-siphash-dev does not have any Built-Using header.
> golang-gopkg-tylerb-graceful.v1-dev does not list golang-x-text,
> although its dependency golang-golang-x-net-dev was built using it.
> (I'm looking at unstable.)

But you are looking at wrong packages. -dev packages are just sources that 
strictly speaking are not "built" but more like "validated" on build time.

You do not need to re-build source/-dev packages so they do not have Built-
Using header intentionally. What you need to be looking at is arch:any binary 
packages built from go sources involving multiple libraries.

Examples of Golang executables include docker.io, containerd, etcd, grafana, 
runc, acbuild, docker2aci, influxdb, rkt, consul, fleet, nomad, skydns, 
kubernetes-{node|master|client}, etc.

-- 
All the best,
 Dmitry Smirnov.

---

To swallow and follow, whether old doctrine or new propaganda, is a
weakness still dominating the human mind.
-- Charlotte P. Gilman


signature.asc
Description: This is a digitally signed message part.
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go

[Florian Weimer]

* Michael Hudson-Doyle:

> On 10 July 2016 at 07:39, Florian Weimer  wrote:
>> * Dmitry Smirnov:
>>
>>> On Friday, 8 July 2016 8:53:20 AM AEST Florian Weimer wrote:
 Part of the problem is that we currently lack a decent way to list all
 these reverse dependencies.
>>>
>>> We can get list of all source packages to re-build from reverse build
>>> dependencies. Then it should be possible to filter arch:any
>>> packages to bin-
>>> NMU.
>>>
>>> Alternatively Built-Using field could be of help.
>>
>> We already discussed why this doesn't work with the present state of
>> the metadata.
>
> Do you mean the "B-U is only direct dependencies" problem? That's
> fixed now.

Hmm.  I poked at a few packages, and here is what I found:
golang-siphash-dev does not have any Built-Using header.
golang-gopkg-tylerb-graceful.v1-dev does not list golang-x-text,
although its dependency golang-golang-x-net-dev was built using it.
(I'm looking at unstable.)

Would this be fixable through a mass rebuild?

> Was there something else?

The dependency-generating script does not end up in Built-Using.
I think we discussed including its version at some point.

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go

[Michael Hudson-Doyle]

On 10 July 2016 at 07:39, Florian Weimer  wrote:
> * Dmitry Smirnov:
>
>> On Friday, 8 July 2016 8:53:20 AM AEST Florian Weimer wrote:
>>> Part of the problem is that we currently lack a decent way to list all
>>> these reverse dependencies.
>>
>> We can get list of all source packages to re-build from reverse build
>> dependencies. Then it should be possible to filter arch:any packages to bin-
>> NMU.
>>
>> Alternatively Built-Using field could be of help.
>
> We already discussed why this doesn't work with the present state of
> the metadata.

Do you mean the "B-U is only direct dependencies" problem? That's
fixed now. Was there something else?

Cheers,
mwh

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go

[Michael Hudson-Doyle]

On 14 April 2016 at 19:16, Michael Stapelberg  wrote:
> Thanks for the patch, it’s now merged and uploaded.

Awesome, thanks for that.

> I’d prefer if you could send such patches in a bug report instead of to
> mailing lists which I don’t actively read :).

Noted!

> In fact, I’d say it’s long overdue to make this package team-maintained.

Sounds sensible.

> The repository is already in
> collab-maint, so if you want to make the necessary changes, please just go
> ahead.

I don't think I can commit to collab-maint (I'm not a DD or even a DM yet).

Cheers,
mwh

> With regards to the original post, I think we have the same issue that the
> haskell packaging community has, since they have the same linking model.
> I’ve talked to Joachim Breitner (nomeata) about this a couple years ago and
> he mentioned they have some tooling which addresses the issue in a
> sufficient way.
>
> I’d suggest to tackle the problem the same way for Go, and maybe share some
> tools if applicable. That said, I won’t have time or motivation to do any of
> the work required for this, so volunteers are very welcome.
>
> On Thu, Apr 14, 2016 at 3:08 AM, Michael Hudson-Doyle
>  wrote:
>>
>> On 13 April 2016 at 21:05, Michael Hudson-Doyle
>>  wrote:
>> > On 13 April 2016 at 17:07, Tianon Gravi  wrote:
>> >> On 12 April 2016 at 21:39, Michael Hudson-Doyle
>> >>  wrote:
>> >>> We could do it without 1) and the consequent re-uploading of every go
>> >>> library by using dpkg-query --search a lot, which would be slow I
>> >>> guess, but maybe could be done as a fallback?
>> >>
>> >> I still asking dpkg about file/directory package ownership should be
>> >> our primary means of generating this field -- the metadata that dpkg
>> >> itself tracks about "which package provided
>> >> /usr/share/gocode/src/abc/xyz which I just compiled against" will
>> >> always be correct (due to the fact that it really is the single proper
>> >> source of truth for such information), where some arbitrary metadata
>> >> we add not only clutters up the package metadata as has been
>> >> discussed, but much more importantly will have a tendency to "drift"
>> >> from the truth, which is something that IMO we shouldn't tolerate for
>> >> a field whose primary purpose is knowing when it's necessary to
>> >> rebuild, especially for security fixes.  Even for really large
>> >> packages like Docker (to choose an example that I know off the top of
>> >> my head is reasonably hefty WRT deps) we're only talking about maybe
>> >> ~200 of these queries at the outside end, and only at build-time, and
>> >> only once per build, which IMO is in the realm of reasonable to avoid
>> >> yet again uploading a minor fix to every package (moving the metadata
>> >> over to the binary packages when we still haven't added the existing
>> >> source package metadata to all of them yet) with information that will
>> >> have a potential for drifting from the truth or for being too limited
>> >> (single package providing multiple namespaces after a repo move, for
>> >> example).
>> >
>> > Yes, all that seems fair. Something like this?
>> > http://paste.ubuntu.com/15806327/ -- it's pretty terrible perl, but
>> > it's actually arguably simpler than what dh_golang does already!
>>
>> FWIW, I sent a better version of this patch:
>>
>> http://lists.alioth.debian.org/pipermail/pkg-go-maintainers/Week-of-Mon-20160411/004304.html
>>
>> Cheers,
>> mwh
>
>
>
>
> --
> Best regards,
> Michael

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go

[Michael Stapelberg]

Thanks for the patch, it’s now merged and uploaded.

I’d prefer if you could send such patches in a bug report instead of to
mailing lists which I don’t actively read :). In fact, I’d say it’s long
overdue to make this package team-maintained. The repository is already in
collab-maint, so if you want to make the necessary changes, please just go
ahead.

With regards to the original post, I think we have the same issue that the
haskell packaging community has, since they have the same linking model.
I’ve talked to Joachim Breitner (nomeata) about this a couple years ago and
he mentioned they have some tooling which addresses the issue in a
sufficient way.

I’d suggest to tackle the problem the same way for Go, and maybe share some
tools if applicable. That said, I won’t have time or motivation to do any
of the work required for this, so volunteers are very welcome.

On Thu, Apr 14, 2016 at 3:08 AM, Michael Hudson-Doyle <
michael.hud...@canonical.com> wrote:

> On 13 April 2016 at 21:05, Michael Hudson-Doyle
>  wrote:
> > On 13 April 2016 at 17:07, Tianon Gravi  wrote:
> >> On 12 April 2016 at 21:39, Michael Hudson-Doyle
> >>  wrote:
> >>> We could do it without 1) and the consequent re-uploading of every go
> >>> library by using dpkg-query --search a lot, which would be slow I
> >>> guess, but maybe could be done as a fallback?
> >>
> >> I still asking dpkg about file/directory package ownership should be
> >> our primary means of generating this field -- the metadata that dpkg
> >> itself tracks about "which package provided
> >> /usr/share/gocode/src/abc/xyz which I just compiled against" will
> >> always be correct (due to the fact that it really is the single proper
> >> source of truth for such information), where some arbitrary metadata
> >> we add not only clutters up the package metadata as has been
> >> discussed, but much more importantly will have a tendency to "drift"
> >> from the truth, which is something that IMO we shouldn't tolerate for
> >> a field whose primary purpose is knowing when it's necessary to
> >> rebuild, especially for security fixes.  Even for really large
> >> packages like Docker (to choose an example that I know off the top of
> >> my head is reasonably hefty WRT deps) we're only talking about maybe
> >> ~200 of these queries at the outside end, and only at build-time, and
> >> only once per build, which IMO is in the realm of reasonable to avoid
> >> yet again uploading a minor fix to every package (moving the metadata
> >> over to the binary packages when we still haven't added the existing
> >> source package metadata to all of them yet) with information that will
> >> have a potential for drifting from the truth or for being too limited
> >> (single package providing multiple namespaces after a repo move, for
> >> example).
> >
> > Yes, all that seems fair. Something like this?
> > http://paste.ubuntu.com/15806327/ -- it's pretty terrible perl, but
> > it's actually arguably simpler than what dh_golang does already!
>
> FWIW, I sent a better version of this patch:
>
> http://lists.alioth.debian.org/pipermail/pkg-go-maintainers/Week-of-Mon-20160411/004304.html
>
> Cheers,
> mwh
>



-- 
Best regards,
Michael
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go

[Michael Hudson-Doyle]

On 13 April 2016 at 21:05, Michael Hudson-Doyle
 wrote:
> On 13 April 2016 at 17:07, Tianon Gravi  wrote:
>> On 12 April 2016 at 21:39, Michael Hudson-Doyle
>>  wrote:
>>> We could do it without 1) and the consequent re-uploading of every go
>>> library by using dpkg-query --search a lot, which would be slow I
>>> guess, but maybe could be done as a fallback?
>>
>> I still asking dpkg about file/directory package ownership should be
>> our primary means of generating this field -- the metadata that dpkg
>> itself tracks about "which package provided
>> /usr/share/gocode/src/abc/xyz which I just compiled against" will
>> always be correct (due to the fact that it really is the single proper
>> source of truth for such information), where some arbitrary metadata
>> we add not only clutters up the package metadata as has been
>> discussed, but much more importantly will have a tendency to "drift"
>> from the truth, which is something that IMO we shouldn't tolerate for
>> a field whose primary purpose is knowing when it's necessary to
>> rebuild, especially for security fixes.  Even for really large
>> packages like Docker (to choose an example that I know off the top of
>> my head is reasonably hefty WRT deps) we're only talking about maybe
>> ~200 of these queries at the outside end, and only at build-time, and
>> only once per build, which IMO is in the realm of reasonable to avoid
>> yet again uploading a minor fix to every package (moving the metadata
>> over to the binary packages when we still haven't added the existing
>> source package metadata to all of them yet) with information that will
>> have a potential for drifting from the truth or for being too limited
>> (single package providing multiple namespaces after a repo move, for
>> example).
>
> Yes, all that seems fair. Something like this?
> http://paste.ubuntu.com/15806327/ -- it's pretty terrible perl, but
> it's actually arguably simpler than what dh_golang does already!

FWIW, I sent a better version of this patch:
http://lists.alioth.debian.org/pipermail/pkg-go-maintainers/Week-of-Mon-20160411/004304.html

Cheers,
mwh

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go

[Michael Hudson-Doyle]

On 13 April 2016 at 17:07, Tianon Gravi  wrote:
> On 12 April 2016 at 21:39, Michael Hudson-Doyle
>  wrote:
>> We could do it without 1) and the consequent re-uploading of every go
>> library by using dpkg-query --search a lot, which would be slow I
>> guess, but maybe could be done as a fallback?
>
> I still asking dpkg about file/directory package ownership should be
> our primary means of generating this field -- the metadata that dpkg
> itself tracks about "which package provided
> /usr/share/gocode/src/abc/xyz which I just compiled against" will
> always be correct (due to the fact that it really is the single proper
> source of truth for such information), where some arbitrary metadata
> we add not only clutters up the package metadata as has been
> discussed, but much more importantly will have a tendency to "drift"
> from the truth, which is something that IMO we shouldn't tolerate for
> a field whose primary purpose is knowing when it's necessary to
> rebuild, especially for security fixes.  Even for really large
> packages like Docker (to choose an example that I know off the top of
> my head is reasonably hefty WRT deps) we're only talking about maybe
> ~200 of these queries at the outside end, and only at build-time, and
> only once per build, which IMO is in the realm of reasonable to avoid
> yet again uploading a minor fix to every package (moving the metadata
> over to the binary packages when we still haven't added the existing
> source package metadata to all of them yet) with information that will
> have a potential for drifting from the truth or for being too limited
> (single package providing multiple namespaces after a repo move, for
> example).

Yes, all that seems fair. Something like this?
http://paste.ubuntu.com/15806327/ -- it's pretty terrible perl, but
it's actually arguably simpler than what dh_golang does already!

Cheers,
mwh

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] [pkg-golang-devel] Security support for packages written in Go

[Paul Tagliamonte]

I don't think B-U is the appropriate place for this. This means if we
didn't change anything in dh-golang, we'd need to binNMU the package before
we can decruft the sources that have a newer versions, dak side.

With an ftp hat on, I think that's not right. Having the entire build
closure in it would be useful, but B-U is also used by dak to keep sources
we still have binaries related to in the archive.

We could add it as some sort of binary control header, but that's also
annoying. Less annoying, though.

On Wed, Apr 6, 2016 at 1:33 PM, Florian Weimer  wrote:

> * Tianon Gravi:
>
> > On 5 April 2016 at 14:47, Florian Weimer  wrote:
> >> We currently need these intermediate dependencies to discover all the
> >> affected applications.  So perhaps dh_golang needs to construct the
> >> transitive closure, instead of listing just immediate build
> >> dependencies.  If we don't want to put this information into the
> >> Packages file, maybe we can keep it in the separate debuginfo
> >> packages.
> >
> > It _should_ be possible to adjust dh_golang to use "go list" in order
> > to determine the exact full set of Go packages which the application
> > code depends on, and then use _that_ list to cross-reference the files
> > in /usr/share/gocode to get the real list of packages for Built-Using
> > ( haven't verified whether it's feasible for dh_golang to do this, but
> > it's pretty similar to how it's currently using "go list" to gather
> > the list of packages to actually build).
>
> Please also add the version of the dh-golang package, so that we know
> what to rebuild if there's a bug in the Built-Using generation.
>
> ___
> Pkg-go-maintainers mailing list
> Pkg-go-maintainers@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
>



-- 
:wq
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers