Re: Possibilities for a special Azure or cloud Linux package

[Bastian Blank]

Hi

On Sat, Nov 18, 2017 at 01:48:53AM +, Ben Hutchings wrote:
> However, if it is possible to create a single flavour that provides
> those sorts of enhancements for multiple cloud platforms, I think that
> would be worthwhile.

I commited a inital config to master.  It is tested on Azure and I'll
take a look on the other clouds later.

commit 5f83961cb506956f9b72c408d3f22ea48f14d0b4 (HEAD -> master, origin/master, 
origin/HEAD)
Author: Bastian Blank 
Date:   Thu Jan 18 09:15:53 2018 +0100

Add cloud-amd64 kernel flavour

As discussed on d-kernel, this flavour is added as experiment on request
of Microsoft.  For now it is only tested on Microsoft Azure.

It will be expanded to cover the other public cloud platforms at well.
This platforms will need additional drivers.

http://lists.alioth.debian.org/pipermail/kernel-svn-changes/2018-January/027184.html

Please take a look if you disapprove with the things I disabled.

Regards,
Bastian

-- 
In the strict scientific sense we all feed on death -- even vegetarians.
-- Spock, "Wolf in the Fold", stardate 3615.4

Re: Possibilities for a special Azure or cloud Linux package

[Noah Meyerhans]

On Fri, Dec 15, 2017 at 08:03:51PM +0100, Bastian Blank wrote:
> > > We at credativ are responsible for maintaining the Azure cloud images.
> > > We got asked by Microsoft to explore the possibilities of introducing a
> > > specialised Linux image for this plattform into Debian.  The main
> > > enhancements we look at would be:
> > > - faster boot of the instance,
> > > - smaller memory footprint of the running kernel, and
> > > - new features.
> > 
> > However, if it is possible to create a single flavour that provides
> > those sorts of enhancements for multiple cloud platforms, I think that
> > would be worthwhile.
> 
> I have some initial findings for a kernel using a derived config.  I
> reduced the boot time by 5 seconds (from 30 to 25).  The installed size
> was reduced from 190 to 50MB.
> 
> Microsoft published a patch set against 4.13 and 4.14
> https://github.com/Microsoft/azure-linux-kernel
> they would like to add.
> 
> Now the question is if other cloud providers would like to follow such a
> path by using and careing for a specialised linux image for this
> platforms?

I'd be interested in helping to support work for a cloud kernel and
verifying its functionality on EC2. I can't, however, make a lot of
promises about how much time I can commit to this effort.

In our previous thread on this topic[1] it was suggested[2] that a tuned
initramfs config might go a long way toward reducing boot times. It's
not an investigation to which I've been able to devote much time,
unfortunately, but I think we should pursue that path to completion
before we look at patching the kernel or providing custom builds. Has
this been done on any cloud platform yet?

noah

1. https://lists.debian.org/debian-cloud/2017/08/msg00025.html
2. https://lists.debian.org/debian-cloud/2017/08/msg00032.html


signature.asc
Description: PGP signature

Re: Possibilities for a special Azure or cloud Linux package

[Thomas Goirand]

On 12/15/2017 08:03 PM, Bastian Blank wrote:
> Now the question is if other cloud providers would like to follow such a
> path by using and careing for a specialised linux image for this
> platforms?
> 
> Bastian

Bastian,

I also raised this topic a few months ago, highlighting the fact that
Ubuntu has such a kernel for virtual machine / cloud. The thread
unfortunately went to nowhere, though if such a kernel flavor was made,
I'd happily use it. The only thing is: I do *not* have time to work on
this, and I do not intend to volunteer / commit into it myself.

Cheers,

Thomas Goirand (zigo)

Re: Possibilities for a special Azure or cloud Linux package

[Bastian Blank]

Hi

I forgot to add the cloud team early to this mail.

On Sat, Nov 18, 2017 at 01:48:53AM +, Ben Hutchings wrote:
> On Fri, 2017-11-17 at 16:07 +0100, Bastian Blank wrote:
> > Hi kernel team
> > 
> > We at credativ are responsible for maintaining the Azure cloud images.
> > We got asked by Microsoft to explore the possibilities of introducing a
> > specialised Linux image for this plattform into Debian.  The main
> > enhancements we look at would be:
> > - faster boot of the instance,
> > - smaller memory footprint of the running kernel, and
> > - new features.
> 
> However, if it is possible to create a single flavour that provides
> those sorts of enhancements for multiple cloud platforms, I think that
> would be worthwhile.

I have some initial findings for a kernel using a derived config.  I
reduced the boot time by 5 seconds (from 30 to 25).  The installed size
was reduced from 190 to 50MB.

Microsoft published a patch set against 4.13 and 4.14
https://github.com/Microsoft/azure-linux-kernel
they would like to add.

Now the question is if other cloud providers would like to follow such a
path by using and careing for a specialised linux image for this
platforms?

Bastian

-- 
Landru! Guide us!
-- A Beta 3-oid, "The Return of the Archons", stardate 3157.4

Re: Possibilities for a special Azure or cloud Linux package

[Bastian Blank]

On Sat, Nov 18, 2017 at 01:48:53AM +, Ben Hutchings wrote:
> We've spent the last decade slowly moving away from platform-specific
> to generic multiplatform kernels on ARM (and, to a lesser extent, x86).
>  I strongly disagree with adding a new flavour that would be specific
> to one vendor's platform.

This is what I also told them.

> However, if it is possible to create a single flavour that provides
> those sorts of enhancements for multiple cloud platforms, I think that
> would be worthwhile.

Okay.  I will explore this variant and see if it provides the benefits
we seek.

> Those are basically the same criteria we normally use, so I don't
> understand your desire to add a featureset.

The people from Microsoft are a bit woried about introducing patches
that break stuff for others except the Azure platform.  E.g. support
for non-Azure HyperV targets or hardware that they use in a bit weird
ways.  So it makes QA easier by limiting the effects of the patches to a
pretty limited set of system configurations.  We should be able to
automatically do tests on those in the future.

Bastian

-- 
Time is fluid ... like a river with currents, eddies, backwash.
-- Spock, "The City on the Edge of Forever", stardate 3134.0

Re: Possibilities for a special Azure or cloud Linux package

[Ben Hutchings]

On Fri, 2017-11-17 at 16:07 +0100, Bastian Blank wrote:
> Hi kernel team
> 
> We at credativ are responsible for maintaining the Azure cloud images.
> We got asked by Microsoft to explore the possibilities of introducing a
> specialised Linux image for this plattform into Debian.  The main
> enhancements we look at would be:
> - faster boot of the instance,
> - smaller memory footprint of the running kernel, and
> - new features.

We've spent the last decade slowly moving away from platform-specific
to generic multiplatform kernels on ARM (and, to a lesser extent, x86).
 I strongly disagree with adding a new flavour that would be specific
to one vendor's platform.

However, if it is possible to create a single flavour that provides
those sorts of enhancements for multiple cloud platforms, I think that
would be worthwhile.

> We don't want to introduce a new source package as was done for
> grsecurity.  This may work, as Microsoft mainly targets the latest
> kernel, which is not always nice.
> 
> That's why I would like to add such an image as a featureset to the main
> Linux package.  This will allow us to look at the downsides of the
> approach in terms of maintenance overhead, build time, archive space
> etc.  If it does not work: disabling a featureset is easy.
> 
> As first step we will not add patches, so the only thing that can break
> is the config.  But just in case I'll propose the following rules for
> patches:
> - must be in Linus tree,
> - must be in a subsystem maintainers tree, or
> - are in linux-next (not sure about this one).

Those are basically the same criteria we normally use, so I don't
understand your desire to add a featureset.

Ben.

> Would it be okay with you if we try this?  Do you see other problem with
> such an approach?
> 
> Bastian
> 
-- 
Ben Hutchings
Teamwork is essential - it allows you to blame someone else.



signature.asc
Description: This is a digitally signed message part

Possibilities for a special Azure or cloud Linux package

[Bastian Blank]

Hi kernel team

We at credativ are responsible for maintaining the Azure cloud images.
We got asked by Microsoft to explore the possibilities of introducing a
specialised Linux image for this plattform into Debian.  The main
enhancements we look at would be:
- faster boot of the instance,
- smaller memory footprint of the running kernel, and
- new features.

We don't want to introduce a new source package as was done for
grsecurity.  This may work, as Microsoft mainly targets the latest
kernel, which is not always nice.

That's why I would like to add such an image as a featureset to the main
Linux package.  This will allow us to look at the downsides of the
approach in terms of maintenance overhead, build time, archive space
etc.  If it does not work: disabling a featureset is easy.

As first step we will not add patches, so the only thing that can break
is the config.  But just in case I'll propose the following rules for
patches:
- must be in Linus tree,
- must be in a subsystem maintainers tree, or
- are in linux-next (not sure about this one).

Would it be okay with you if we try this?  Do you see other problem with
such an approach?

Bastian

-- 
Most legends have their basis in facts.
-- Kirk, "And The Children Shall Lead", stardate 5029.5