Re: my progress

[Thomas Lange]

> On Sun, 13 Nov 2016 06:42:37 -0800, Noah Meyerhans  
> said:

> I based my work on jessie initially because we have JEB's high quality
> jessie images to use as a baseline for comparison, and I found that
> useful.
+1

We agreed that we want to use fai for the official Debian images from
stretch. But the users may want to use the new tool for their
customized jessie images, or give it a try before they have to use it
for stretch anyway.

IMO it was a good idea to start with jessie and we should keep this
config in the official git repository.
-- 
regards Thomas

Re: my progress

[Noah Meyerhans]

On Sun, Nov 13, 2016 at 06:17:33AM -0500, Sam Hartman wrote:
> Noah> I've modified the class/DEBIAN.var file such that the default
> Noah> behavior is to generate images for jessie. We can add a
> Noah> STRETCH class in order to generate images for testing. I'd
> Noah> rather use "stable", but it's preferable to use the release
> Noah> codename in many places and there's no easy way to resolve
> Noah> stable to the current codename AFAIK. Maybe there should be?
> 
> I thought we agreed the fai efforts would be for stretch and we would
> not be converting our existing jessie work.
> I don't mind having jessie support if it's easy, but for example if we
> run into situations where  it adds extra code to work around bugs fixed
> in stretch but present in jessie, I'd like to push back.

Yes, I think you're right.

I based my work on jessie initially because we have JEB's high quality
jessie images to use as a baseline for comparison, and I found that
useful.

In practice, activating a JESSIE or STRETCH class still needs to happen
via the fai-diskimage commandline anyway, and these classes override the
release variable, so defining a default value for that variable in
class/DEBIAN.var was not actually useful. My inclination is to require
the user to activate a specific release class, and for FAI to abort if
one isn't provided.

noah




signature.asc
Description: PGP signature

Re: my progress

[Sam Hartman]

> "Noah" == Noah Meyerhans  writes:

Noah> I've modified the class/DEBIAN.var file such that the default
Noah> behavior is to generate images for jessie. We can add a
Noah> STRETCH class in order to generate images for testing. I'd
Noah> rather use "stable", but it's preferable to use the release
Noah> codename in many places and there's no easy way to resolve
Noah> stable to the current codename AFAIK. Maybe there should be?

I thought we agreed the fai efforts would be for stretch and we would
not be converting our existing jessie work.
I don't mind having jessie support if it's easy, but for example if we
run into situations where  it adds extra code to work around bugs fixed
in stretch but present in jessie, I'd like to push back.

I suspect that long-term find it easier to maintain something where we
have the work-arounds for things we've found in stable releases in a
class, and the main body of the code is targeted at sid and testing.
Thoughts?

Re: my progress

[Noah Meyerhans]

And I just pushed some changes to introduce basic stretch support via a
STRETCH class. The following commandline resulted in a working image for
EC2:

fai-diskimage -u stretch8g -S8G -cDEBIAN,STRETCH,AMD64,GRUB_PC,DEVEL,CLOUD,EC2 
~/disk.raw

I haven't done anything to optimize the configuration for stretch, but
the image does boot and the resulting instance properly sets up login
access via cloud-init.



signature.asc
Description: PGP signature

Re: my progress

[Noah Meyerhans]

On Sun, Nov 06, 2016 at 11:38:33AM -0800, Noah Meyerhans wrote:
> > I pushed to git://git.debian.org/cloud/fai-cloud-images.git.
> 
> I've got a FAI config targeting jessie on EC2. It seems to work well and
> at this point I'm at the point of tweaking the packages lists and
> configuration to match JEB's images as closely as possible. It shouldn't
> be hard to merge my work into Sam's repo, but I haven't done it yet.

Ok, I've just pushed to fai-cloud-images. Because my work started in a
different repo, it was difficult to preserve history, so I ended up
pushing everything in a single commit.

My volume-to-ami script is included at the top level. It's certainly not
the right place for it, but there isn't an obvious right place. Maybe
create a bin directory?

In an effort to reduce repetition, I've introduced an "apt_cdn" variable
that can be set in a cloud-specific class. This lets us generate
sources.list files based on templates that refer to mirrors running in
the local cloud infrastructure, e.g. cloudfront for AWS.

I've modified the class/DEBIAN.var file such that the default behavior
is to generate images for jessie. We can add a STRETCH class in order to
generate images for testing. I'd rather use "stable", but it's
preferable to use the release codename in many places and there's no
easy way to resolve stable to the current codename AFAIK. Maybe there
should be?

I've generated ami-113e9d71 based on the following command line:
fai-diskimage -u cloud3 -S3G 
-cDEBIAN,JESSIE,JESSIE64,AMD64,BACKPORTS,GRUB_PC,DEVEL,CLOUD,EC2

That AMI is marked public in case anybody wants to try it as is, though
I may delete it at some point.

cloud-init works, along with root FS resize. Console output is logged to
the serial port and is available via the get-console-output API. The
'admin' user is created with the ssh key given at boot time. pstree
shows:

systemd-+-agetty
|-agetty
|-atd
|-cron
|-dbus-daemon
|-dhclient
|-exim4
|-rsyslogd-+-{in:imklog}
|  |-{in:imuxsock}
|  `-{rs:main Q:Reg}
|-sshd---sshd---sshd---bash---pstree
|-systemd-journal
|-systemd-logind
`-systemd-udevd

Systemd-analyze critical-chain ssh.service shows that, between kernel
and userspace, it takes 40s before the instance is ready for use, so
it'd be nice to optimize this:

ssh.service @36.153s
└─cloud-init.service @27.088s +9.040s
  └─cloud-init-local.service @12.938s +14.136s
└─basic.target @12.938s
  └─timers.target @12.938s
└─systemd-tmpfiles-clean.timer @12.938s
  └─sysinit.target @12.937s
└─networking.service @6.533s +6.403s
  └─local-fs.target @6.532s
└─mdadm-raid.service @6.112s +419ms
  └─systemd-udevd.service @6.099s +3ms
└─systemd-tmpfiles-setup-dev.service @5.353s +745ms
  └─kmod-static-nodes.service @4.642s +710ms
└─system.slice @4.640s
  └─-.slice @4.640s


These values compare favorably to our current AWS AMIs, although there
are some notable differences that should be addressed. In particular, my
image runs exim4 and atd, while the marketplace AMI does not. We
probably should stick with the marketplace AMI's behavior for
consistency. My image fixes 785457 by not running extraneous gettys on
tty1-6.

My AMI has 550 packages installed, while the marketplace AMI has only
318. I have not compared image generation when performed without the
DEVEL class being enabled. I suspect it'll still have more packages than
marketplace, but not by as wide a margin.

noah



signature.asc
Description: PGP signature

Re: my progress

[Sam Hartman]

> "Noah" == Noah Meyerhans  writes:

Noah> On Fri, Nov 04, 2016 at 09:03:39PM -0400, Sam Hartman wrote:
>> I pushed to git://git.debian.org/cloud/fai-cloud-images.git.

Noah> It looks like you're the only person with write access to this
Noah> repo. It seems like changing group ownership to 'scm_cloud'
Noah> and granting group write permission is the right fix.

Fixed, my bad.

Re: my progress

[Martin Zobel-Helas]

No, i am not, this is alioth. Dsa is not maintaining alioth.

Am 11. November 2016 17:27:15 MEZ, schrieb Noah Meyerhans :
>On Fri, Nov 04, 2016 at 09:03:39PM -0400, Sam Hartman wrote:
>> I pushed to git://git.debian.org/cloud/fai-cloud-images.git.
>
>It looks like you're the only person with write access to this repo. It
>seems like changing group ownership to 'scm_cloud' and granting group
>write permission is the right fix.
>
>zobel, are you able to help with this?
>
>noah

-- 
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

Re: my progress

[Noah Meyerhans]

On Fri, Nov 04, 2016 at 09:03:39PM -0400, Sam Hartman wrote:
> I pushed to git://git.debian.org/cloud/fai-cloud-images.git.

It looks like you're the only person with write access to this repo. It
seems like changing group ownership to 'scm_cloud' and granting group
write permission is the right fix.

zobel, are you able to help with this?

noah




signature.asc
Description: PGP signature

Re: AWS build workflow (was Re: my progress)

[Noah Meyerhans]

On Mon, Nov 07, 2016 at 08:23:10AM +, Marcin Kulisz wrote:
> That's true but there is a bit which makes me quite uncomfortable to be
> precise it's that to do all this stuff from within Debian infra we need to 
> keep
> AWS IAM keys on it with permissions for spinning up and down instances etc.

Yes. The keys would need to be associated with a role granted access to
the following API calls:

* Run instance
* Describe instance
* Create volume
* Attachh volume
* Create snapshot
* Describe snapshot
* Register AMI
* Terminate instance

I'm not sure what facilities are provided on debian.org machines for
managing access to sensitive material such as IAM credentials in an
automated way.

> From my conversation with JEB kind of vision emerged that we could have
> combination of api gateway and lambda listening on the api point and those
> would spin up instance with Pettersson ssh key (public part ofc) and specific
> IAM role on it to allow to do DD and all AWS related dance. Once whole process
> is done it'll just destroy AWS instance and wait for the next build.
> Clean and neat use of "the cloud" I'd say.

My recollection from the sprint is that we agreed that we'd like to
build the images on official Debian infrastructure to the extent
possible, which is why I proposed that workflow. However, I agree that
there are alternatives that make use of some of the other AWS services
such as Lambda, KMS, etc.

noah



signature.asc
Description: PGP signature

Re: my progress

[Martin Zobel-Helas]

Hi, 

On Sun Nov 06, 2016 at 11:38:33 -0800, Noah Meyerhans wrote:
> On Fri, Nov 04, 2016 at 09:03:39PM -0400, Sam Hartman wrote:
> > I pushed to git://git.debian.org/cloud/fai-cloud-images.git.
> 
> I've got a FAI config targeting jessie on EC2. It seems to work well and
> at this point I'm at the point of tweaking the packages lists and
> configuration to match JEB's images as closely as possible. It shouldn't
> be hard to merge my work into Sam's repo, but I haven't done it yet.
> 
> I've introduced two classes:
> 
> EC2: Contains EC2-specific packages, debconf settings, etc
> 
> DEVEL: Installs a number of packages useful in a full-featured Debian
> environment, such as git, devscripts, build-essential, etc. The
> motivation is that we could generate "minimal" images simply by leaving
> omitting this class.

While above packages would be expected by Debian Developers, users will
probably not expect those packages in a normal EC2 images.

Cheers,
Martin
-- 
 Martin Zobel-Helas Debian System Administrator
 Debian & GNU/Linux Developer   Debian Listmaster
 http://about.me/zobel   Debian Webmaster
 GPG Fingerprint:  6B18 5642 8E41 EC89 3D5D  BDBB 53B1 AC6D B11B 627B 

AWS build workflow (was Re: my progress)

[Marcin Kulisz]

On 2016-11-06 11:38:33, Noah Meyerhans wrote:
> On Fri, Nov 04, 2016 at 09:03:39PM -0400, Sam Hartman wrote:
> > I pushed to git://git.debian.org/cloud/fai-cloud-images.git.
> 
> I've got a FAI config targeting jessie on EC2. It seems to work well and
> at this point I'm at the point of tweaking the packages lists and
> configuration to match JEB's images as closely as possible. It shouldn't
> be hard to merge my work into Sam's repo, but I haven't done it yet.
> 
> I've introduced two classes:
> 
> EC2: Contains EC2-specific packages, debconf settings, etc
> 
> DEVEL: Installs a number of packages useful in a full-featured Debian
> environment, such as git, devscripts, build-essential, etc. The
> motivation is that we could generate "minimal" images simply by leaving
> omitting this class.

+1

> My workflow has largely involved local development with testing taking
> place in EC2. In reality there's only a single step that needs to be
> performed on an actual instance, though. ('dd' of the generated image to
> an attached EBS volume) A possible "production" workflow might be:
> 
> 1. Run 'fai-diskimage' locally on a Debian-managed host.
> 2. Perform automated analysis of FAI logs for sanity tests.
> 3. Mount generated image locally and perform filesystem-level tests.
> 4. Launch an EC2 instance and attach an EBS volume to it as sdb.
> 5. scp the generated image to the EC2 instance and 'dd' to sdb.
> 6. snapshot the EBS volume and register it as an AMI
> 7. Perform any desired AMI tests, such as launching it with different
> user-data configuration, etc.
> 8. Migrate the AMI to different regions, mark public, register with
> marketplace, etc.
>
> The 'dd' component of step 5 is the only one that actually needs to run
> on an EC2 instance.

That's true but there is a bit which makes me quite uncomfortable to be
precise it's that to do all this stuff from within Debian infra we need to keep
AWS IAM keys on it with permissions for spinning up and down instances etc.

From my conversation with JEB kind of vision emerged that we could have
combination of api gateway and lambda listening on the api point and those
would spin up instance with Pettersson ssh key (public part ofc) and specific
IAM role on it to allow to do DD and all AWS related dance. Once whole process
is done it'll just destroy AWS instance and wait for the next build.
Clean and neat use of "the cloud" I'd say.

> I'm sure JEB already has infrastructure for most of these steps, and we
> should continue using that where appropriate.

From what I know we (me and JEB) have 2 independent build nodes and we're
planning to converge into the above at least that's how I'm understanding
conclusions of our conversations.

But looking at what you wrote as a high level overview matches exactly what we
(most of us in Seattle I think) agreed upon including me.

So thx for your work on AWS part and pls push it to the git, hopefully Mrfai
will fix fai to work on LVM based hosts so I can try it :-)
-- 

|_|0|_|  |
|_|_|0|  "Panta rei" |
|0|0|0|  kuLa    |

gpg --keyserver pgp.mit.edu --recv-keys 0x686930DD58C338B3
3DF1  A4DF  C732  4688  38BC  F121  6869  30DD  58C3  38B3


signature.asc
Description: PGP signature

Re: my progress

[Sam Hartman]

Your proposed workflow is one I've used with live-build for generating
EC2 images for years.
It works well.

Re: my progress

[Thomas Lange]

> On Sun, 6 Nov 2016 11:38:33 -0800, Noah Meyerhans  said:

> 3. Mount generated image locally and perform filesystem-level tests.
Hi Noah,

this can be done inside FAI itself. You can put some scripts into the
tests/ subdirectory and name them after a class, see fai-do-scripts(1)
for details. Then they will be executed. The scripts can access the
still mounted target directory which is available in the variable
$target. For example a test may look like this (not tested at all):

#! /bin/bash

error=0; trap 'error=$(($?>$error?$?:$error))' ERR # save maximum error code

if [ ! -f $target/boot/grub/grub.cfg ]; then
   echo "TEST FAILED. No grub.cfg available"
   task_error 812
fi

exit $error


The function task_error will set a special variable which is checked
against $STOP_ON_ERROR, which is default to 700. All error codes
greater this value should stop the FAI installation. It also displays
a red icon on the fai-monitor-gui, if it's running.


-- 
regards Thomas

Re: my progress

[Noah Meyerhans]

On Fri, Nov 04, 2016 at 09:03:39PM -0400, Sam Hartman wrote:
> I pushed to git://git.debian.org/cloud/fai-cloud-images.git.

I've got a FAI config targeting jessie on EC2. It seems to work well and
at this point I'm at the point of tweaking the packages lists and
configuration to match JEB's images as closely as possible. It shouldn't
be hard to merge my work into Sam's repo, but I haven't done it yet.

I've introduced two classes:

EC2: Contains EC2-specific packages, debconf settings, etc

DEVEL: Installs a number of packages useful in a full-featured Debian
environment, such as git, devscripts, build-essential, etc. The
motivation is that we could generate "minimal" images simply by leaving
omitting this class.

My workflow has largely involved local development with testing taking
place in EC2. In reality there's only a single step that needs to be
performed on an actual instance, though. ('dd' of the generated image to
an attached EBS volume) A possible "production" workflow might be:

1. Run 'fai-diskimage' locally on a Debian-managed host.
2. Perform automated analysis of FAI logs for sanity tests.
3. Mount generated image locally and perform filesystem-level tests.
4. Launch an EC2 instance and attach an EBS volume to it as sdb.
5. scp the generated image to the EC2 instance and 'dd' to sdb.
6. snapshot the EBS volume and register it as an AMI
7. Perform any desired AMI tests, such as launching it with different
user-data configuration, etc.
8. Migrate the AMI to different regions, mark public, register with
marketplace, etc.

The 'dd' component of step 5 is the only one that actually needs to run
on an EC2 instance.

I'm sure JEB already has infrastructure for most of these steps, and we
should continue using that where appropriate.

noah



signature.asc
Description: PGP signature

Re: my progress

[Thomas Lange]

Hi Sam,

today we've managed to produced an GCE image which was also tested with
success. It boots, and we can log into this machine. We had to do some
changes in the config space. Everything is now available in the
examples of FAI on github. I plan to make a new 
FAI release in the middle of next week.

-- 
regards Thomas