Bug#971515: Request for security team input on kubernetes TC bug

[Florian Weimer]

* Moritz Mühlenhoff:

> On Sun, Nov 08, 2020 at 10:49:31PM +0100, Florian Weimer wrote:
>> * Moritz Mühlenhoff:
>> 
>> > * Follow a scheme similar to Firefox ESR where in case of a security
>> >   the update either happens to the latest minor release of
>> >   the current branch or if that has stopped, happens to the next
>> >   major release. To map this to specific k8s releases: Let's assume 
>> > bullseye
>> >   were already stable and would ship 19.3. In three months a security issue
>> >   arises which is severe enough to warrant an update and we ship 19.5
>> >   in a DSA. Fast forward 6 months and 19.x is EOLed, but some severe
>> >   security issue needs to be fixed; then the bullseye update would move
>> >   to 20.1 or so. That sounds unusual for a Debian release, but it's
>> >   the status quo for _any_ Kubernetes user after all (whether deployed
>> >   on premises or at some "cloud vendor").
>> 
>> Another thing to consider: Kubernetes rebases will likely require Go
>> rebases, if I read this table correctly:
>> 
>> 
>
> I can't tell how strict these requirements are in practice.

Let's just say that some Kubernetes developers are very eager to get
their hands on the most recent Go toolchain even if there are
practical issues with choices in the run-time library, such as the
changes to certification validation.  Not sure if anyone would want to
suffer these toolchain rebases if there was a clean way around them. 8-)

> It's similar to Firefox requiring more recent versions of rustc and
> golang packages are co-installable, so when it comes to that, shipping
> a new golang-x.y package might also be an option.

I see.

>> Since Go has a bit of spotty history in terms of kernel backwards
>> compatibility, this could cause further challenges.
>
> Do you have a concrete example of such a change? I see that 1.14 is
> available in backports, so this seems to work out so far.

I think that's it: 
If I recall correctly, there was a kernel version check (!) that
managed to reject kernels which did not have the bug.  And combined
with the workaround failing, this led to non-functional programs.

Bug#971515: Request for security team input on kubernetes TC bug

[Moritz Mühlenhoff]

On Sun, Nov 08, 2020 at 10:49:31PM +0100, Florian Weimer wrote:
> * Moritz Mühlenhoff:
> 
> > * Follow a scheme similar to Firefox ESR where in case of a security
> >   the update either happens to the latest minor release of
> >   the current branch or if that has stopped, happens to the next
> >   major release. To map this to specific k8s releases: Let's assume bullseye
> >   were already stable and would ship 19.3. In three months a security issue
> >   arises which is severe enough to warrant an update and we ship 19.5
> >   in a DSA. Fast forward 6 months and 19.x is EOLed, but some severe
> >   security issue needs to be fixed; then the bullseye update would move
> >   to 20.1 or so. That sounds unusual for a Debian release, but it's
> >   the status quo for _any_ Kubernetes user after all (whether deployed
> >   on premises or at some "cloud vendor").
> 
> Another thing to consider: Kubernetes rebases will likely require Go
> rebases, if I read this table correctly:
> 
> 

I can't tell how strict these requirements are in practice.

It's similar to Firefox requiring more recent versions of rustc and
golang packages are co-installable, so when it comes to that, shipping
a new golang-x.y package might also be an option.

> Since Go has a bit of spotty history in terms of kernel backwards
> compatibility, this could cause further challenges.

Do you have a concrete example of such a change? I see that 1.14 is available
in backports, so this seems to work out so far.

Cheers,
Moritz

Bug#971515: Request for security team input on kubernetes TC bug

[Moritz Mühlenhoff]

Catching up on this...

> > This leaves Debian with two options:
> > * Keep it out of a stable release and accept that it's good enough
> >   if people just install whatever deb they currently find in testing/sid
> >   (works out well enough for most given that blob nature of Go!)
> 
> IMHO this is the most reasonable option and perhaps the only viable one.

Ultimately that's for the release team and/or CTTE to make a call on.

And this while discussion also needs the input of Janos as the current 
kubernetes
maintainer.

> > * Follow a scheme similar to Firefox ESR where in case of a security
> >   the update either happens to the latest minor release of
> >   the current branch or if that has stopped, happens to the next
> >   major release.
> 
> I think Kubernetes have many more vendored 3rd party libraries than Firefox.
> IMHO we can not expect the same level of confidence for Kubernetes...

But each of their releases constitutes a bundle of third party libraries
they've vetted to work together, so this seems to work in practice? (as
can be seen by the 1.19 upload from today). Or maybe I'm missing the specific
concern of yours, is this about them missing fixes in their bundled libs?

Cheers,
Moritz

Bug#971515: Request for security team input on kubernetes TC bug

[Florian Weimer]

* Moritz Mühlenhoff:

> * Follow a scheme similar to Firefox ESR where in case of a security
>   the update either happens to the latest minor release of
>   the current branch or if that has stopped, happens to the next
>   major release. To map this to specific k8s releases: Let's assume bullseye
>   were already stable and would ship 19.3. In three months a security issue
>   arises which is severe enough to warrant an update and we ship 19.5
>   in a DSA. Fast forward 6 months and 19.x is EOLed, but some severe
>   security issue needs to be fixed; then the bullseye update would move
>   to 20.1 or so. That sounds unusual for a Debian release, but it's
>   the status quo for _any_ Kubernetes user after all (whether deployed
>   on premises or at some "cloud vendor").

Another thing to consider: Kubernetes rebases will likely require Go
rebases, if I read this table correctly:



Since Go has a bit of spotty history in terms of kernel backwards
compatibility, this could cause further challenges.

Bug#971515: Request for security team input on kubernetes TC bug

[Dmitry Smirnov]

On Wednesday, 28 October 2020 6:13:41 AM AEDT Moritz Mühlenhoff wrote:
> The bigger issue here (independent of the whole vendoring aspect) is
> how kubernetes can be supported in a stable release to begin with.
> This was raised by Shengjing Zhu in #959685 before.

If Kubernetes can be supported then such support will be done by upstream,
but with extraordinary amount of dependencies (and upstream reluctance to 
manage them), I have very low confidence and low expectations for quality
of such support. The problem primarily is that Kubernetes vendors hundreds
of dependencies representing a large support surface. Effectively it is
"#include world" (or vendor world) situation. And when it comes to problems
in 3rd party vendored libraries, it iw worth remembering that Kubernetes 
don't own them.


> This leaves Debian with two options:
> * Keep it out of a stable release and accept that it's good enough
>   if people just install whatever deb they currently find in testing/sid
>   (works out well enough for most given that blob nature of Go!)

IMHO this is the most reasonable option and perhaps the only viable one.


> * Follow a scheme similar to Firefox ESR where in case of a security
>   the update either happens to the latest minor release of
>   the current branch or if that has stopped, happens to the next
>   major release.

I think Kubernetes have many more vendored 3rd party libraries than Firefox.
IMHO we can not expect the same level of confidence for Kubernetes...

-- 
Best wishes,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

You have to start with the truth. The truth is the only way that we can
get anywhere. Because any decision-making that is based upon lies or
ignorance can't lead to a good conclusion.
-- Julian Assange, 2010


signature.asc
Description: This is a digitally signed message part.

Bug#971515: Request for security team input on kubernetes TC bug

[Moritz Mühlenhoff]

On Wed, Oct 21, 2020 at 08:22:11AM -0700, Sean Whitton wrote:
> Hello security team,
> 
> The TC are being asked about src:kubernetes, and it would be good to
> hear from you about whether and how security support is a relevant
> consideration in determining whether the level of vendoring in that
> package is acceptable.  Could you take a look at #971515 and perhaps
> give us some input, please?

I don't currently have the time to read through all the discussion backlog,
but let me try to "zoom out a bit":

The bigger issue here (independent of the whole vendoring aspect) is
how kubernetes can be supported in a stable release to begin with.
This was raised by Shengjing Zhu in #959685 before.

Upstream support only lasts for up to a year and it would be
presumptuous to pretend that we can seriously commit to fix
security issues in k8s for longer than upstream.

This leaves Debian with two options:
* Keep it out of a stable release and accept that it's good enough
  if people just install whatever deb they currently find in testing/sid
  (works out well enough for most given that blob nature of Go!)
* Follow a scheme similar to Firefox ESR where in case of a security
  the update either happens to the latest minor release of
  the current branch or if that has stopped, happens to the next
  major release. To map this to specific k8s releases: Let's assume bullseye
  were already stable and would ship 19.3. In three months a security issue
  arises which is severe enough to warrant an update and we ship 19.5
  in a DSA. Fast forward 6 months and 19.x is EOLed, but some severe
  security issue needs to be fixed; then the bullseye update would move
  to 20.1 or so. That sounds unusual for a Debian release, but it's
  the status quo for _any_ Kubernetes user after all (whether deployed
  on premises or at some "cloud vendor").

If we follow the latter approach, then the current way k8s is packaged
is the only viable option (as otherwise the run time deps shipped
for 19.x would be incompatible with 20.x etc.)

Cheers,
Moritz

Bug#971515: Request for security team input on kubernetes TC bug

[Dmitry Smirnov]

On Thursday, 22 October 2020 2:22:11 AM AEDT Sean Whitton wrote:
> The TC are being asked about src:kubernetes, and it would be good to
> hear from you about whether and how security support is a relevant
> consideration in determining whether the level of vendoring in that
> package is acceptable.  Could you take a look at #971515 and perhaps
> give us some input, please?

I'm not a member of security team but I want to share my prospective on this.

To us what matters is what we can do about vulnerability in situation where
the problem has been identified and fixed upstream, in the particular 
library.

When a library is packaged separately, this is a matter of patching/updating 
a particular library then re-building depending packages.

With vendored libraries, we have no control as patching arbitrary versions of 
all instances of vendored library is a very bad option that is practically 
unsustainable. Even tracking of vendored libraries is difficult.

So in case of vendored libraries we can rely on "Oracle model" when upstream 
releases an update of a software where vendored library is patched so we ship 
the whole bundle. This is the worst option because we have no control and 
because we have to rely on faith in good upstream maintenance.

But here lies the problem. Kubernetes have bad dependency hygiene and poor 
abolity to control their enormous vendor mess (burden). With hundreds of 
vendored libraries they have large surface for problems and limited ability 
to track and respond. Just like we know from experience, an attempt to update 
a vendored library may lead to FTBFS and that's why upstream is reluctant to 
make changes to vendored libraries, unless when they absolutely have to.
Kubernetes can not effectively address issues in "vendor/" space.

So my conclusion about security support of upstream Kubernetes is that it is 
can not be done effectively in either case. There are greater chances for 
meaningful security maintenance when most dependency libraries are un-
vendored. With most libraries vendored, there is less security maintenance 
burden to us, but the outcome is no better.

-- 
All the best,
 Dmitry Smirnov
 GPG key : 4096R/52B6BBD953968D1B

---

No snowflake in an avalanche ever feels responsible.
-- Stanisław Jerzy Lec


signature.asc
Description: This is a digitally signed message part.

Bug#971515: Request for security team input on kubernetes TC bug

[Sean Whitton]

Hello security team,

The TC are being asked about src:kubernetes, and it would be good to
hear from you about whether and how security support is a relevant
consideration in determining whether the level of vendoring in that
package is acceptable.  Could you take a look at #971515 and perhaps
give us some input, please?

Thanks.

-- 
Sean Whitton