Re: [pkg-go] Security support for packages written in Go

[Moritz Muehlenhoff]

On Wed, Oct 12, 2016 at 11:27:28AM +1300, Michael Hudson-Doyle wrote:
> On 12 October 2016 at 10:44, Moritz Mühlenhoff  wrote:
> 
> > On Mon, Jul 11, 2016 at 10:53:57AM +1200, Michael Hudson-Doyle wrote:
> > > On 9 July 2016 at 07:21, Moritz Muehlenhoff  wrote:
> > > > Florian Weimer wrote:
> > > >> > On Wednesday, 6 July 2016 9:59:32 PM AEST Moritz Mühlenhoff wrote:
> > > >> >> What's the current status? Is there technical progress compared to
> > what was
> > > >> >> discussed in April? The freeze is coming really close and we can't
> > support
> > > >> >> the status quo for stretch.
> > > >> >
> > > >> > Perhaps I'm not the best person to speak on the matter as I've never
> > > >> > touched any Golang tools except dh-golang. Situation with Golab
> > > >> > libraries is not ideal (to say the least) but I understand that
> > > >> > Golang is not the only language without concept of dynamic
> > > >> > linking. As I recall someone mentioned Haskell as another example.
> > > >> >
> > > >> > It is my understanding that when vulnerability is fixed in Golang
> > > >> > library it should be sufficient to NMU (re-build) all reverse
> > > >> > dependencies.
> > > >>
> > > >> Part of the problem is that we currently lack a decent way to list all
> > > >> these reverse dependencies.
> > > >
> > > > And there's also the much bigger problem that we can't actually rebuild
> > > > packages on security.debian.org without a lot of manual work!
> > > >
> > > > The dak installation for security-master has a _lot_ of tech debt. One
> > > > that particularly bites us here is that tarballs between
> > security-master
> > > > and ftp-master are separate. This e.g. requires that every package that
> > > > is new on security-master needs to be build with "-sa" to include
> > source
> > > > and we can only issue binNMUs for packages which were at least once
> > > > upload to jessie-security/stretch-security etc.
> > >
> > > That does sound unfortunate in the Go context.
> > >
> > > It is worth bearing in mind though, that you only need to rebuild the
> > > binary-containing packages, so if the number of binary-containing
> > > packages supported by the security team is tightly constrained, then
> > > so is the number of (no-change source, I guess) uploads required to
> > > handle any security update (e.g. in Ubuntu 16.04 there are only three
> > > packages that contain Go binaries in main).
> > >
> > > The changes I'm making in Ubuntu to use shared libraries should in the
> > > common case (i.e. the fix does not break ABI) make this better, but
> > > worst case (where the fix breaks ABI) it will be worse as we might end
> > > up having to rebuild the whole rdep tree.
> >
> > What's the status of that work, will that land in the stretch release?
> >
> 
> I'm not planning on working on getting it into stretch myself.

What does it entail to get it in stretch, is it part of a specific Go upstream 
release?

Cheers,
Moritz

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Michael Hudson-Doyle]

On 12 October 2016 at 10:44, Moritz Mühlenhoff  wrote:

> On Mon, Jul 11, 2016 at 10:53:57AM +1200, Michael Hudson-Doyle wrote:
> > On 9 July 2016 at 07:21, Moritz Muehlenhoff  wrote:
> > > Florian Weimer wrote:
> > >> > On Wednesday, 6 July 2016 9:59:32 PM AEST Moritz Mühlenhoff wrote:
> > >> >> What's the current status? Is there technical progress compared to
> what was
> > >> >> discussed in April? The freeze is coming really close and we can't
> support
> > >> >> the status quo for stretch.
> > >> >
> > >> > Perhaps I'm not the best person to speak on the matter as I've never
> > >> > touched any Golang tools except dh-golang. Situation with Golab
> > >> > libraries is not ideal (to say the least) but I understand that
> > >> > Golang is not the only language without concept of dynamic
> > >> > linking. As I recall someone mentioned Haskell as another example.
> > >> >
> > >> > It is my understanding that when vulnerability is fixed in Golang
> > >> > library it should be sufficient to NMU (re-build) all reverse
> > >> > dependencies.
> > >>
> > >> Part of the problem is that we currently lack a decent way to list all
> > >> these reverse dependencies.
> > >
> > > And there's also the much bigger problem that we can't actually rebuild
> > > packages on security.debian.org without a lot of manual work!
> > >
> > > The dak installation for security-master has a _lot_ of tech debt. One
> > > that particularly bites us here is that tarballs between
> security-master
> > > and ftp-master are separate. This e.g. requires that every package that
> > > is new on security-master needs to be build with "-sa" to include
> source
> > > and we can only issue binNMUs for packages which were at least once
> > > upload to jessie-security/stretch-security etc.
> >
> > That does sound unfortunate in the Go context.
> >
> > It is worth bearing in mind though, that you only need to rebuild the
> > binary-containing packages, so if the number of binary-containing
> > packages supported by the security team is tightly constrained, then
> > so is the number of (no-change source, I guess) uploads required to
> > handle any security update (e.g. in Ubuntu 16.04 there are only three
> > packages that contain Go binaries in main).
> >
> > The changes I'm making in Ubuntu to use shared libraries should in the
> > common case (i.e. the fix does not break ABI) make this better, but
> > worst case (where the fix breaks ABI) it will be worse as we might end
> > up having to rebuild the whole rdep tree.
>
> What's the status of that work, will that land in the stretch release?
>

I'm not planning on working on getting it into stretch myself.

Cheers,
mwh
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Moritz Mühlenhoff]

On Mon, Jul 11, 2016 at 10:53:57AM +1200, Michael Hudson-Doyle wrote:
> On 9 July 2016 at 07:21, Moritz Muehlenhoff  wrote:
> > Florian Weimer wrote:
> >> > On Wednesday, 6 July 2016 9:59:32 PM AEST Moritz Mühlenhoff wrote:
> >> >> What's the current status? Is there technical progress compared to what 
> >> >> was
> >> >> discussed in April? The freeze is coming really close and we can't 
> >> >> support
> >> >> the status quo for stretch.
> >> >
> >> > Perhaps I'm not the best person to speak on the matter as I've never
> >> > touched any Golang tools except dh-golang. Situation with Golab
> >> > libraries is not ideal (to say the least) but I understand that
> >> > Golang is not the only language without concept of dynamic
> >> > linking. As I recall someone mentioned Haskell as another example.
> >> >
> >> > It is my understanding that when vulnerability is fixed in Golang
> >> > library it should be sufficient to NMU (re-build) all reverse
> >> > dependencies.
> >>
> >> Part of the problem is that we currently lack a decent way to list all
> >> these reverse dependencies.
> >
> > And there's also the much bigger problem that we can't actually rebuild
> > packages on security.debian.org without a lot of manual work!
> >
> > The dak installation for security-master has a _lot_ of tech debt. One
> > that particularly bites us here is that tarballs between security-master
> > and ftp-master are separate. This e.g. requires that every package that
> > is new on security-master needs to be build with "-sa" to include source
> > and we can only issue binNMUs for packages which were at least once
> > upload to jessie-security/stretch-security etc.
> 
> That does sound unfortunate in the Go context.
> 
> It is worth bearing in mind though, that you only need to rebuild the
> binary-containing packages, so if the number of binary-containing
> packages supported by the security team is tightly constrained, then
> so is the number of (no-change source, I guess) uploads required to
> handle any security update (e.g. in Ubuntu 16.04 there are only three
> packages that contain Go binaries in main).
> 
> The changes I'm making in Ubuntu to use shared libraries should in the
> common case (i.e. the fix does not break ABI) make this better, but
> worst case (where the fix breaks ABI) it will be worse as we might end
> up having to rebuild the whole rdep tree.

What's the status of that work, will that land in the stretch release?

Cheers,
Moritz

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Michael Hudson-Doyle]

On 13 July 2016 at 19:20, Moritz Mühlenhoff  wrote:
> On Mon, Jul 11, 2016 at 09:12:05AM +1200, Michael Hudson-Doyle wrote:
>> On 8 July 2016 at 20:03, Potter, Tim (HPE Linux Support)
>>  wrote:
>> > On 7 Jul 2016, at 12:40 PM, Martín Ferrari  wrote:
>> >>
>> >> On 06/07/16 20:59, Moritz Mühlenhoff wrote:
>> >>
>> >>> What's the current status? Is there technical progress compared to what 
>> >>> was
>> >>> discussed in April? The freeze is coming really close and we can't 
>> >>> support
>> >>> the status quo for stretch.
>> >>
>> >> The discussion stalled at that point. AFAIK, there is no technical
>> >> solution for this. The best we could do is to have easier ways to track
>> >> dependency chains, but that does not change the fact that all golang
>> >> applications are still statically built, and so would require rebuilds
>> >> when security bugs are discovered and fixed.
>> >>
>> >> I understand this is problematic, but not sure we can do anything about
>> >> it at this point.
>> >
>> > Hi everyone.  I've done a small amount of research into the 
>> > buildmode=c-shared
>> > and the dynlink option and they look good on paper.  Has anyone examined 
>> > these
>> > options more seriously?
>>
>> Well, using them in Ubuntu was the reason Canonical paid me to
>> implement them, so yes... I'm am currently in the process of starting
>> to use these features in Ubuntu. My plan, such as it was, was to use
>> them in Ubuntu through the 16.10 cycle and then propose the changes to
>> Debian too, assuming they work out OK.
>
> What does the provide specifically? Dynamic linking similar to what we 
> currently
> have for library code written in C?

Yes. There are more details here:
https://docs.google.com/document/d/1IOlBWWgcDeB9PfRORENESYj8iJt4W2EwsbYcpg4akBE/edit

Cheers,
mwh

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Moritz Mühlenhoff]

On Mon, Jul 11, 2016 at 09:12:05AM +1200, Michael Hudson-Doyle wrote:
> On 8 July 2016 at 20:03, Potter, Tim (HPE Linux Support)
>  wrote:
> > On 7 Jul 2016, at 12:40 PM, Martín Ferrari  wrote:
> >>
> >> On 06/07/16 20:59, Moritz Mühlenhoff wrote:
> >>
> >>> What's the current status? Is there technical progress compared to what 
> >>> was
> >>> discussed in April? The freeze is coming really close and we can't support
> >>> the status quo for stretch.
> >>
> >> The discussion stalled at that point. AFAIK, there is no technical
> >> solution for this. The best we could do is to have easier ways to track
> >> dependency chains, but that does not change the fact that all golang
> >> applications are still statically built, and so would require rebuilds
> >> when security bugs are discovered and fixed.
> >>
> >> I understand this is problematic, but not sure we can do anything about
> >> it at this point.
> >
> > Hi everyone.  I've done a small amount of research into the 
> > buildmode=c-shared
> > and the dynlink option and they look good on paper.  Has anyone examined 
> > these
> > options more seriously?
> 
> Well, using them in Ubuntu was the reason Canonical paid me to
> implement them, so yes... I'm am currently in the process of starting
> to use these features in Ubuntu. My plan, such as it was, was to use
> them in Ubuntu through the 16.10 cycle and then propose the changes to
> Debian too, assuming they work out OK.

What does the provide specifically? Dynamic linking similar to what we currently
have for library code written in C?

Cheers,
Moritz

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Michael Hudson-Doyle]

On 9 July 2016 at 07:21, Moritz Muehlenhoff  wrote:
> Florian Weimer wrote:
>> > On Wednesday, 6 July 2016 9:59:32 PM AEST Moritz Mühlenhoff wrote:
>> >> What's the current status? Is there technical progress compared to what 
>> >> was
>> >> discussed in April? The freeze is coming really close and we can't support
>> >> the status quo for stretch.
>> >
>> > Perhaps I'm not the best person to speak on the matter as I've never
>> > touched any Golang tools except dh-golang. Situation with Golab
>> > libraries is not ideal (to say the least) but I understand that
>> > Golang is not the only language without concept of dynamic
>> > linking. As I recall someone mentioned Haskell as another example.
>> >
>> > It is my understanding that when vulnerability is fixed in Golang
>> > library it should be sufficient to NMU (re-build) all reverse
>> > dependencies.
>>
>> Part of the problem is that we currently lack a decent way to list all
>> these reverse dependencies.
>
> And there's also the much bigger problem that we can't actually rebuild
> packages on security.debian.org without a lot of manual work!
>
> The dak installation for security-master has a _lot_ of tech debt. One
> that particularly bites us here is that tarballs between security-master
> and ftp-master are separate. This e.g. requires that every package that
> is new on security-master needs to be build with "-sa" to include source
> and we can only issue binNMUs for packages which were at least once
> upload to jessie-security/stretch-security etc.

That does sound unfortunate in the Go context.

It is worth bearing in mind though, that you only need to rebuild the
binary-containing packages, so if the number of binary-containing
packages supported by the security team is tightly constrained, then
so is the number of (no-change source, I guess) uploads required to
handle any security update (e.g. in Ubuntu 16.04 there are only three
packages that contain Go binaries in main).

The changes I'm making in Ubuntu to use shared libraries should in the
common case (i.e. the fix does not break ABI) make this better, but
worst case (where the fix breaks ABI) it will be worse as we might end
up having to rebuild the whole rdep tree.

Cheers,
mwh

> And with that setup (and in addition to what Florian mentioned) I see
> no sane way that we can support Go applications in stretch. It's
> already difficult enough to support a distro of the size of Debian with
> volunteers only.

> As Haskell was mentioned; sure it has the same problem. But Go is a totally
> different ballpark: Go aims at system-level services which have a notable
> security surface (think of docker or kubernetes), while I can't remember
> any security vulnerability in an application written in Haskell in the
> archive in the 10+ years I'm in the Security Team.
>
> Cheers,
> Moritz
>
> ___
> Pkg-go-maintainers mailing list
> Pkg-go-maintainers@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Michael Hudson-Doyle]

On 8 July 2016 at 20:03, Potter, Tim (HPE Linux Support)
 wrote:
> On 7 Jul 2016, at 12:40 PM, Martín Ferrari  wrote:
>>
>> On 06/07/16 20:59, Moritz Mühlenhoff wrote:
>>
>>> What's the current status? Is there technical progress compared to what was
>>> discussed in April? The freeze is coming really close and we can't support
>>> the status quo for stretch.
>>
>> The discussion stalled at that point. AFAIK, there is no technical
>> solution for this. The best we could do is to have easier ways to track
>> dependency chains, but that does not change the fact that all golang
>> applications are still statically built, and so would require rebuilds
>> when security bugs are discovered and fixed.
>>
>> I understand this is problematic, but not sure we can do anything about
>> it at this point.
>
> Hi everyone.  I've done a small amount of research into the buildmode=c-shared
> and the dynlink option and they look good on paper.  Has anyone examined these
> options more seriously?

Well, using them in Ubuntu was the reason Canonical paid me to
implement them, so yes... I'm am currently in the process of starting
to use these features in Ubuntu. My plan, such as it was, was to use
them in Ubuntu through the 16.10 cycle and then propose the changes to
Debian too, assuming they work out OK.

> My guess would be that there might be a bunch of as yet undiscovered
> platform-specific bugs with this,

FWIW, I currently know of no platform specific bugs (if you use 1.7rc1
or the patches on 1.6.2 I have in Ubuntu). But I have been in this
position before :-)

> and it's probably too late at this stage to make
> such a major change to the golang package build process.

I would tend to agree that getting this stood up in time for stretch
isn't a great idea.

Cheers,
mwh

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Martín Ferrari]

On 09/07/16 20:39, Florian Weimer wrote:

>> We can get list of all source packages to re-build from reverse build 
>> dependencies. Then it should be possible to filter arch:any packages to bin-
>> NMU.
>>
>> Alternatively Built-Using field could be of help.
> 
> We already discussed why this doesn't work with the present state of
> the metadata.

while the current state prevents having properly automated tools
covering all cases, we already have quite a bit of information in that
field, and we can always manually discover and re-upload applications
when a rdep has a security fix. I know it is not ideal, but also it does
not prevent us from providing proper support, IMHO. 

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Florian Weimer]

* Dmitry Smirnov:

> On Friday, 8 July 2016 8:53:20 AM AEST Florian Weimer wrote:
>> Part of the problem is that we currently lack a decent way to list all
>> these reverse dependencies.
>
> We can get list of all source packages to re-build from reverse build 
> dependencies. Then it should be possible to filter arch:any packages to bin-
> NMU.
>
> Alternatively Built-Using field could be of help.

We already discussed why this doesn't work with the present state of
the metadata.

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Potter, Tim (HPE Linux Support)]

On 9 Jul 2016, at 3:52 PM, Martín Ferrari  wrote:
> 
> Moritz,
> 
> On 08/07/16 20:21, Moritz Muehlenhoff wrote:

>> And with that setup (and in addition to what Florian mentioned) I see
>> no sane way that we can support Go applications in stretch. It's
>> already difficult enough to support a distro of the size of Debian with
>> volunteers only.
> 
> So what would be the way forward for this? Declaring golang apps to be
> not supported for security updates? I am ready to live with that,
> although I know I will do any work needed to prepare security fixes for
> the few apps I maintain.

A declaration of non-support for security updates for golang apps is not my
preferred solution either.  (-:

I'm willing to talk with our security team at HPE to work out something here
for the stretch release though.


Tim.

> 
> --
> Martín Ferrari (Tincho)
> 
> ___
> Pkg-go-maintainers mailing list
> Pkg-go-maintainers@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Martín Ferrari]

Moritz,

On 08/07/16 20:21, Moritz Muehlenhoff wrote:

> And there's also the much bigger problem that we can't actually rebuild
> packages on security.debian.org without a lot of manual work!
> 
> The dak installation for security-master has a _lot_ of tech debt. One
> that particularly bites us here is that tarballs between security-master
> and ftp-master are separate. This e.g. requires that every package that
> is new on security-master needs to be build with "-sa" to include source
> and we can only issue binNMUs for packages which were at least once
> upload to jessie-security/stretch-security etc.

I understand the pain here. But again, I don't think we can do anything
to fix that. And in this case, seems to me that the effort should be
aimed at fixing that technical debt.

> And with that setup (and in addition to what Florian mentioned) I see
> no sane way that we can support Go applications in stretch. It's
> already difficult enough to support a distro of the size of Debian with
> volunteers only.

So what would be the way forward for this? Declaring golang apps to be
not supported for security updates? I am ready to live with that,
although I know I will do any work needed to prepare security fixes for
the few apps I maintain.

-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Ian Campbell]

On Fri, 2016-07-08 at 21:21 +0200, Moritz Muehlenhoff wrote:
> 
> As Haskell was mentioned; sure it has the same problem. But Go is a totally
> different ballpark:

I mentioned Haskell only because AIUI they have a tool for generating
sets of binNMUs from a changed source package, I didn't intend to imply
"haskell is ok ergo go must be too" or anything like that, sorry for
not being clear on that.

Ian.


___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Moritz Muehlenhoff]

Florian Weimer wrote:
> > On Wednesday, 6 July 2016 9:59:32 PM AEST Moritz Mühlenhoff wrote:
> >> What's the current status? Is there technical progress compared to what was
> >> discussed in April? The freeze is coming really close and we can't support
> >> the status quo for stretch.
> >
> > Perhaps I'm not the best person to speak on the matter as I've never
> > touched any Golang tools except dh-golang. Situation with Golab
> > libraries is not ideal (to say the least) but I understand that
> > Golang is not the only language without concept of dynamic
> > linking. As I recall someone mentioned Haskell as another example.
> >
> > It is my understanding that when vulnerability is fixed in Golang
> > library it should be sufficient to NMU (re-build) all reverse
> > dependencies.
> 
> Part of the problem is that we currently lack a decent way to list all
> these reverse dependencies.

And there's also the much bigger problem that we can't actually rebuild
packages on security.debian.org without a lot of manual work!

The dak installation for security-master has a _lot_ of tech debt. One
that particularly bites us here is that tarballs between security-master
and ftp-master are separate. This e.g. requires that every package that
is new on security-master needs to be build with "-sa" to include source
and we can only issue binNMUs for packages which were at least once
upload to jessie-security/stretch-security etc.

And with that setup (and in addition to what Florian mentioned) I see
no sane way that we can support Go applications in stretch. It's
already difficult enough to support a distro of the size of Debian with
volunteers only.

As Haskell was mentioned; sure it has the same problem. But Go is a totally
different ballpark: Go aims at system-level services which have a notable
security surface (think of docker or kubernetes), while I can't remember
any security vulnerability in an application written in Haskell in the
archive in the 10+ years I'm in the Security Team.

Cheers,
Moritz

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Potter, Tim (HPE Linux Support)]

On 7 Jul 2016, at 12:40 PM, Martín Ferrari  wrote:
> 
> On 06/07/16 20:59, Moritz Mühlenhoff wrote:
> 
>> What's the current status? Is there technical progress compared to what was
>> discussed in April? The freeze is coming really close and we can't support
>> the status quo for stretch.
> 
> The discussion stalled at that point. AFAIK, there is no technical
> solution for this. The best we could do is to have easier ways to track
> dependency chains, but that does not change the fact that all golang
> applications are still statically built, and so would require rebuilds
> when security bugs are discovered and fixed.
> 
> I understand this is problematic, but not sure we can do anything about
> it at this point.

Hi everyone.  I've done a small amount of research into the buildmode=c-shared
and the dynlink option and they look good on paper.  Has anyone examined these
options more seriously?

My guess would be that there might be a bunch of as yet undiscovered
platform-specific bugs with this, and it's probably too late at this stage to 
make
such a major change to the golang package build process.


Tim.


signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Martín Ferrari]

On 06/07/16 20:59, Moritz Mühlenhoff wrote:

> What's the current status? Is there technical progress compared to what was
> discussed in April? The freeze is coming really close and we can't support
> the status quo for stretch.

The discussion stalled at that point. AFAIK, there is no technical
solution for this. The best we could do is to have easier ways to track
dependency chains, but that does not change the fact that all golang
applications are still statically built, and so would require rebuilds
when security bugs are discovered and fixed.

I understand this is problematic, but not sure we can do anything about
it at this point.


-- 
Martín Ferrari (Tincho)

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Moritz Mühlenhoff]

On Wed, Apr 06, 2016 at 09:24:20AM +1000, Dmitry Smirnov wrote:
> IMHO Golang community abused almost as much as possible with static linking, 
> embedding resources to executables, not using versioning and breaking API at 
> any time, etc.
> 
> Even if we find effective technical solution to detect dependency chains 
> there is a problem of re-building ever growing number of all packages 
> indirectly depending on vulnerable package.
> 
> Golang is just too young, unstable and fragile. I have a feeling that few 
> upstream projects take security concerns seriously. Many upstream projects 
> have no concept of "stable" releases so I doubt it is practical to offer any 
> kind of security support for Golang when too many projects introduce new 
> dependencies with almost every new versioned release while old release is 
> abandoned as soon as new one becomes available.
> 
> Unless we can exclude Golang from security support I think we should not ship 
> any Golang applications with next stable release.

What's the current status? Is there technical progress compared to what was
discussed in April? The freeze is coming really close and we can't support
the status quo for stretch.

Cheers,
Moritz



___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Michael Hudson-Doyle]

On 13 April 2016 at 17:03, Florian Weimer  wrote:
> * Michael Hudson-Doyle:
>
>> There is another approach to the static linking issue, which is to
>> start using dynamic linking instead. It's implemented upstream for
>> most architectures now (only mips64 le/be and ppc64 be missing I
>> think). I'm going to be working on starting to use dynamic linking
>> during the next cycle of Ubuntu development, and I'd certainly be
>> interested in getting it going for Debian too. (the timeframes re:
>> stretch release look reasonable for this).
>
> Can you explain a bit more how dynamic linking would help us to
> determine what we need to rebuild?

Well, some of the time, rebuilding won't be needed, hopefully.

Also, the way my prototype dh-golang change works, a libgolang*
package Provides a value that contains the abi hash and dependencies
depend on the hash value (via dpkg-shlibdeps), so in that case
figuring out how much to rebuild is a case of "build stuff until
britney stops shouting at you about making packages uninstallable" (I
don't know if that's practical for the way you build security updates
though).

> I expect that dynamic linking will complicate matters because we will
> have to rebuild library packages in dependency order.  I don't see how
> Go shared objects can provide a stable ABI.

Over releases, no, I think you're right, but I really hope that
security fixes can at least sometimes preserve ABI (the crypto fixes
in Go 1.6.1 would not break ABI, for example).

Cheers,
mwh

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Florian Weimer]

* Michael Hudson-Doyle:

> There is another approach to the static linking issue, which is to
> start using dynamic linking instead. It's implemented upstream for
> most architectures now (only mips64 le/be and ppc64 be missing I
> think). I'm going to be working on starting to use dynamic linking
> during the next cycle of Ubuntu development, and I'd certainly be
> interested in getting it going for Debian too. (the timeframes re:
> stretch release look reasonable for this).

Can you explain a bit more how dynamic linking would help us to
determine what we need to rebuild?

I expect that dynamic linking will complicate matters because we will
have to rebuild library packages in dependency order.  I don't see how
Go shared objects can provide a stable ABI.

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Peter Colberg]

On Tue, Apr 05, 2016 at 06:05:21PM -0400, Paul Tagliamonte wrote:
> Love this idea, I wonder if the Import-Path XS header could help resolve
> packages in a proof of concept

If I am not mistaken, the XS-Go-Import-Path cannot be queried with
dpkg-query since it is a field in the source package. What do you
think of adding an XB-Go-Import-Path to golang-*-dev packages?

Querying this field for 'golang-*-dev' to match the import paths
output by `go list` would appear to be a lot faster than invoking
`dpkg -S /usr/share/gocode/src/…` for each import path.

Peter

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Paul Tagliamonte]

We can change it to XSB-Go-Import-Path, but it only really applies for the
-dev packages; so it might need some fiddling to do right, yeah. I'll think
a bit more about it.

We could also likely build up mappings for Source -> import path, and index
from Binary control Source -> Source -> Import Path, but that's a bit
janky, yeah.



On Wed, Apr 6, 2016 at 2:52 PM, Peter Colberg  wrote:

> On Tue, Apr 05, 2016 at 06:05:21PM -0400, Paul Tagliamonte wrote:
> > Love this idea, I wonder if the Import-Path XS header could help resolve
> > packages in a proof of concept
>
> If I am not mistaken, the XS-Go-Import-Path cannot be queried with
> dpkg-query since it is a field in the source package. What do you
> think of adding an XB-Go-Import-Path to golang-*-dev packages?
>
> Querying this field for 'golang-*-dev' to match the import paths
> output by `go list` would appear to be a lot faster than invoking
> `dpkg -S /usr/share/gocode/src/…` for each import path.
>
> Peter
>
> ___
> Pkg-go-maintainers mailing list
> Pkg-go-maintainers@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
>



-- 
:wq
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Dmitry Smirnov]

On Tuesday, 5 April 2016 10:41:04 PM AEST Paul Tagliamonte wrote:
> | Backports are packages taken from the next Debian release (called
> | "testing"), adjusted and recompiled for usage on Debian stable.
> 
> So my confusion here is that you don't want to see them in Stable, but
> you do want to see them in testing (and backports). This isn't what
> testing is for, of course :)

I'm not comfortable with this idea. It requires a lot more work as well.
I just entertain possibility if we have to drop golang apps from next stable. 
After all even if not in next stable we might have some good ideas for next 
stable+1...


> I don't see anything inherent about Go that makes it unsupportable.

You are certainly more optimistic than I am. :)


> I *do* see more software being developed in a way that makes it nearly
> impossible for Debian to distribute.

But I'm not talking about distribution. We can distribute but security 
support is very difficult. Maybe we should just give up on it...


> This, however, is a much bigger conversation.

True...

-- 
Regards,
 Dmitry Smirnov.

---

Lies are the social equivalent of toxic waste: Everyone is potentially
harmed by their spread.
-- Sam Harris


signature.asc
Description: This is a digitally signed message part.
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Paul Tagliamonte]

On Wed, Apr 06, 2016 at 12:37:10PM +1000, Dmitry Smirnov wrote:
> I feel your pain. Over last 9 months I've invested even greater effort to 
> packaging of containers related Golang software.
> 
> Yet we can provide anything we want to users of stable releases through 
> official backports:
> 
> http://backports.debian.org/

| Backports are packages taken from the next Debian release (called
| "testing"), adjusted and recompiled for usage on Debian stable.

So my confusion here is that you don't want to see them in Stable, but
you do want to see them in testing (and backports). This isn't what
testing is for, of course :)

This, to me, feels like more of a systemic issue than this small side
conversation.

I don't see anything inherent about Go that makes it unsupportable.

I *do* see more software being developed in a way that makes it nearly
impossible for Debian to distribute. This, however, is a much bigger
conversation.

Cheers,
  Paul


signature.asc
Description: PGP signature
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Peter Colberg]

On Wed, Apr 06, 2016 at 09:24:20AM +1000, Dmitry Smirnov wrote:
> Unless we can exclude Golang from security support I think we should not ship 
> any Golang applications with next stable release.

I really hope not, that would be a real shame.

All the work that we did together on acmetool (#817091) would have
been for nothing. The only reason I went through the trouble of
packaging all these Go modules is to have acmetool in the stretch
release so it can be used on stable servers.

Peter

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Dmitry Smirnov]

On Tuesday, 5 April 2016 9:27:27 AM AEST Florian Weimer wrote:
> we need to discuss how we can support applications written in Go for
> stretch.
> 
> The most radical approach would be not to ship any Go applications in
> stretch, only the basic Go language implementations.  This is probably
> not desirable.

Alternatively we could explicitly disclaim security support for Golang 
applications.
IMHO Golang community abused almost as much as possible with static linking, 
embedding resources to executables, not using versioning and breaking API at 
any time, etc.

Even if we find effective technical solution to detect dependency chains 
there is a problem of re-building ever growing number of all packages 
indirectly depending on vulnerable package.

Golang is just too young, unstable and fragile. I have a feeling that few 
upstream projects take security concerns seriously. Many upstream projects 
have no concept of "stable" releases so I doubt it is practical to offer any 
kind of security support for Golang when too many projects introduce new 
dependencies with almost every new versioned release while old release is 
abandoned as soon as new one becomes available.

Unless we can exclude Golang from security support I think we should not ship 
any Golang applications with next stable release.

Unfortunately if we exclude Golang binaries from release Debian will not be 
competitive in the enterprise sector: industry is rapidly developing Golang 
tools for container solutions and they are crucial for Debian success.

To some extent we can compensate for absence of Golang binaries in stable 
release if we continue developing in "testing" and make selected apps 
available through stretch-backports suite.


> One approach would be to ship applications as source code only (just
> like libraries), and compile them locally upon installation.  This is
> what Emacs and Common Lisp implementations already do.  With the
> growth of Go compilation and link times, this seems less and less
> attractive, though.

This is a possible solution but with Golang it is not practical as building 
some packages takes too much time and resources. Also we still need to build 
packages on build servers to have tests coverage and build logs. 
Unfortunately shipping Golang applications in sources only (no binaries) is 
not an option...

-- 
All the best,
 Dmitry Smirnov.

---

Good luck happens when preparedness meets opportunity.


signature.asc
Description: This is a digitally signed message part.
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Tianon Gravi]

https://sources.debian.net/src/dh-golang/1.12/script/dh_golang/#L121
is where Built-Using is added (generated from the code above that
line)

https://sources.debian.net/src/dh-golang/1.12/lib/Debian/Debhelper/Buildsystem/golang.pm/#L144
is where dh-golang currently invokes "go list" (on "DH_GOPKG/..."
which is set from XS-Go-Import-Path or "DH_GOLANG_BUILDPKG" which is
set in d/rules)

http://dave.cheney.net/2014/09/14/go-list-your-swiss-army-knife is a
good overview of what "go list" is capable of ("Who depends on what?"
is the interesting section which talks about -f '{{ .Imports }}' and
-f '{{ .Deps }}')

♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4


On 5 April 2016 at 15:05, Paul Tagliamonte  wrote:
> Love this idea, I wonder if the Import-Path XS header could help resolve
> packages in a proof of concept
>
> On Apr 5, 2016 5:54 PM, "Tianon Gravi"  wrote:
>>
>> On 5 April 2016 at 14:47, Florian Weimer  wrote:
>> > We currently need these intermediate dependencies to discover all the
>> > affected applications.  So perhaps dh_golang needs to construct the
>> > transitive closure, instead of listing just immediate build
>> > dependencies.  If we don't want to put this information into the
>> > Packages file, maybe we can keep it in the separate debuginfo
>> > packages.
>>
>> It _should_ be possible to adjust dh_golang to use "go list" in order
>> to determine the exact full set of Go packages which the application
>> code depends on, and then use _that_ list to cross-reference the files
>> in /usr/share/gocode to get the real list of packages for Built-Using
>> ( haven't verified whether it's feasible for dh_golang to do this, but
>> it's pretty similar to how it's currently using "go list" to gather
>> the list of packages to actually build).
>>
>> ♥,
>> - Tianon
>>   4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4
>>
>> ___
>> Pkg-go-maintainers mailing list
>> Pkg-go-maintainers@lists.alioth.debian.org
>> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Paul Tagliamonte]

Love this idea, I wonder if the Import-Path XS header could help resolve
packages in a proof of concept
On Apr 5, 2016 5:54 PM, "Tianon Gravi"  wrote:

> On 5 April 2016 at 14:47, Florian Weimer  wrote:
> > We currently need these intermediate dependencies to discover all the
> > affected applications.  So perhaps dh_golang needs to construct the
> > transitive closure, instead of listing just immediate build
> > dependencies.  If we don't want to put this information into the
> > Packages file, maybe we can keep it in the separate debuginfo
> > packages.
>
> It _should_ be possible to adjust dh_golang to use "go list" in order
> to determine the exact full set of Go packages which the application
> code depends on, and then use _that_ list to cross-reference the files
> in /usr/share/gocode to get the real list of packages for Built-Using
> ( haven't verified whether it's feasible for dh_golang to do this, but
> it's pretty similar to how it's currently using "go list" to gather
> the list of packages to actually build).
>
> ♥,
> - Tianon
>   4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4
>
> ___
> Pkg-go-maintainers mailing list
> Pkg-go-maintainers@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers
___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Tianon Gravi]

On 5 April 2016 at 14:47, Florian Weimer  wrote:
> We currently need these intermediate dependencies to discover all the
> affected applications.  So perhaps dh_golang needs to construct the
> transitive closure, instead of listing just immediate build
> dependencies.  If we don't want to put this information into the
> Packages file, maybe we can keep it in the separate debuginfo
> packages.

It _should_ be possible to adjust dh_golang to use "go list" in order
to determine the exact full set of Go packages which the application
code depends on, and then use _that_ list to cross-reference the files
in /usr/share/gocode to get the real list of packages for Built-Using
( haven't verified whether it's feasible for dh_golang to do this, but
it's pretty similar to how it's currently using "go list" to gather
the list of packages to actually build).

♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers

Re: [pkg-go] Security support for packages written in Go

[Florian Weimer]

* Martín Ferrari:

>> The alternative is to rebuild reverse dependencies as needed.  I can
>> see two challenges with that.  Right now, the Built-Using field only
>> records the source versions of the *direct* dependencies (based on the
>> dh_golang manual page and a few examples I looked at).  If a critical
>> update happens farther down the dependency chain, a tool based on
>> Built-Using will not mark the top-level package as a rebuild
>> candidate.  When performing the rebuild, it is possible to compensate
>> for that by rebuilding everything that has an outdated Go source
>> package in its Build-Using field, iteratively, until we reach a
>> fixpoint.  But this does not currently work because the -dev packages
>> do not contain Built-Using information.
>
> Actually, I had not noticed this before. I have been including the
> built-using field in control files, assuming it will end in the binary
> package too. Maybe we can try to fix this?
>
> Alternatively, the built-using field could include the closure of all
> transitive dependencies, although that might explode in size...
>
> In any case, we need to take into account that a security fix in a
> library usually will not require security uploads to intermediate
> dependencies.

We currently need these intermediate dependencies to discover all the
affected applications.  So perhaps dh_golang needs to construct the
transitive closure, instead of listing just immediate build
dependencies.  If we don't want to put this information into the
Packages file, maybe we can keep it in the separate debuginfo
packages.

>> It does not seem possible to determine rebuild candidates based on
>> Built-Using alone, building the transitive closure after the fact.  It
>> may have changed between the original application build and subsequent
>> library builds.
>
> What if we build the transitive closure and discarded any arch:all
> binaries from the rebuild?

Anything is possible with dak support.  But it's better not to count
on it and rebuild only what we actually need to ship.

>> Unrelated to all that, we cannot currently perform binNMUs in the
>> security archive because it does not contain a completely copy of the
>> main archive.  I'm not sure if there are good approaches to deal with
>> this yet.
>
> So this would be an argument for keeping the status quo and just
> rebuilding applications manually with a sourceful upload?

Then we absolutely have to minimize what we rebuild.

Keeping the status quo is barely acceptable for this aspect, but there
is no status quo to keep for the discovery of packages which need
rebuilding.  We just don't have that capability right now.

___
Pkg-go-maintainers mailing list
Pkg-go-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-go-maintainers