Re: vmdebootstrap plans

[Lars Wirzenius]

On Sun, Nov 13, 2016 at 08:58:02PM +0100, Thomas Lange wrote:
> I'm really wondering why you want to use YAML for this? Your examples
> look like things you could easily implement in shell, if you provide
> some functions, subroutines or external programs that implement your
> steps.

I don't think using the shell language to implement a DSL is
acceptable, however easy it may seem. Shell is a really, really bad
programming language and I don't want to encourage its use.

In addition, I will _not_ use shell to implement anything that
requires robustness or code clarity, for it is a lousy language for
anything beyond very simple scripts. Life is too short to write,
maintain, or debug, complicated software written in shell.

I'm too old for this sh.

> create_raw_file $size $image

I realize that you only wrote the above as part of an example, but
to me it's an example of why shell won't do. You didn't quote the
variable expansions, which may result in quite nasty bugs. Reading the
code I would need to evaluate if it's safe to not quote, or if it is
in fact correct to not quote, or if there's a possible bug there. The
rules for quoting in shell are too intricate, and it's too easy to
accidentally do the wrong thing.

> Much easier and more common for sysadmins IMO.

Having written this kind of software before, I guarantee it's not
easier in the long term. Or the medium term. It's only easier for
getting to a quick prototype. Beyond that, one runs into language
limitations.

If editing a YAML file is too difficult for a sysadmin, then I don't
really care. I don't think it will be. YAML files are easy and a very
large number of sysadmins use them with, say, Ansible.

-- 
I want to build worthwhile things that might last. --joeyh


signature.asc
Description: PGP signature

Re: vmdebootstrap plans

[Chris Fordham]

On 14 Nov 2016 8:39 am, "Marcin Kulisz"  wrote:
>
> On 2016-11-13 20:58:02, Thomas Lange wrote:
> > Hi Lars,
> >
> > I'm really wondering why you want to use YAML for this?
>
> I can talk just for my self but my pros would be: cleanliness, machine
> parcebility, human redeability, neat python integration, simple templating
> integration (jinja2) with programming languages. So I suppose Lars
> would agree with those and possibly add something else.
>
> > Your examples
> > look like things you could easily implement in shell, if you provide
> > some functions, subroutines or external programs that implement your
> > steps. Then your DSL would just be a shell script that the user has to
> > write.
> >
> >
> > create_raw_file $size $image
> > create_partition_table msdos
> > create_partition $bootsize ext2 /boot bootable
> > create_partition rest ext4 /
> > mount_all
>
> Above doesn't look so neat to me like well structured schema based yaml.
>
> > and so on.
> >
> > >   The "step:" key specifies the step to execute. The other keys
are
> > >   parameters for the step. The {{...}} parts are where Jinja2 is
used
> > >   to expand variable values.
> > These things are very similar to calling commands with parameters in a
> > shell script without the need of YAML and Jinja2.
>
> Yes, but IMO yaml and jinja2 allow for creating very simple and nice
> integration and extending capabilities with minimal effort.
>
> > Much easier and more common for sysadmins IMO.
>
> I can agree that this may be the case for some people but it's hard to
deny
> that there are restrictions on bash which can't be overcome due to bash
nature
> and which does not exist in python as this is fully blown programming
language.
>
> And yes I know Thomas your approach to object orientated programming :-)
>
> What Lars is proposing I see as a very reasonable thing to do. Providing
well design
> and maintained tool with easy extensibility and a way to accommodate
eventual
> future requirements.
>
> IMO what Lars is trying to achieve is much easier to do in python then in
bash
> and with properly abstracted logic there should be no need to anybody
using
> this tool to even know python (what for me is a bit odd for sysadmins) to
create
> images with vmdb.
> Knowledge of available options and basics of yaml would suffice and I
think
> this is even lower threshold for entry level then knowing bash.

These days golang is the go, excuse the pun and the fact it compiles to a
single executable means the language and extra packages such as jinja2 are
not required.
> --
>
> |_|0|_|  |
> |_|_|0|  "Panta rei" |
> |0|0|0|  kuLa    |
>
> gpg --keyserver pgp.mit.edu --recv-keys 0x686930DD58C338B3
> 3DF1  A4DF  C732  4688  38BC  F121  6869  30DD  58C3  38B3

Re: vmdebootstrap plans

[Marcin Kulisz]

On 2016-11-13 20:58:02, Thomas Lange wrote:
> Hi Lars,
> 
> I'm really wondering why you want to use YAML for this?

I can talk just for my self but my pros would be: cleanliness, machine
parcebility, human redeability, neat python integration, simple templating
integration (jinja2) with programming languages. So I suppose Lars
would agree with those and possibly add something else.

> Your examples
> look like things you could easily implement in shell, if you provide
> some functions, subroutines or external programs that implement your
> steps. Then your DSL would just be a shell script that the user has to
> write.
> 
> 
> create_raw_file $size $image
> create_partition_table msdos
> create_partition $bootsize ext2 /boot bootable
> create_partition rest ext4 /
> mount_all

Above doesn't look so neat to me like well structured schema based yaml.

> and so on.
> 
> >   The "step:" key specifies the step to execute. The other keys are
> >   parameters for the step. The {{...}} parts are where Jinja2 is used
> >   to expand variable values.
> These things are very similar to calling commands with parameters in a
> shell script without the need of YAML and Jinja2.

Yes, but IMO yaml and jinja2 allow for creating very simple and nice
integration and extending capabilities with minimal effort.

> Much easier and more common for sysadmins IMO.

I can agree that this may be the case for some people but it's hard to deny
that there are restrictions on bash which can't be overcome due to bash nature
and which does not exist in python as this is fully blown programming language.

And yes I know Thomas your approach to object orientated programming :-)

What Lars is proposing I see as a very reasonable thing to do. Providing well 
design
and maintained tool with easy extensibility and a way to accommodate eventual
future requirements.

IMO what Lars is trying to achieve is much easier to do in python then in bash
and with properly abstracted logic there should be no need to anybody using
this tool to even know python (what for me is a bit odd for sysadmins) to create
images with vmdb.
Knowledge of available options and basics of yaml would suffice and I think
this is even lower threshold for entry level then knowing bash.
-- 

|_|0|_|  |
|_|_|0|  "Panta rei" |
|0|0|0|  kuLa    |

gpg --keyserver pgp.mit.edu --recv-keys 0x686930DD58C338B3
3DF1  A4DF  C732  4688  38BC  F121  6869  30DD  58C3  38B3


signature.asc
Description: PGP signature

Re: my progress

[Thomas Lange]

> On Sun, 13 Nov 2016 06:42:37 -0800, Noah Meyerhans  
> said:

> I based my work on jessie initially because we have JEB's high quality
> jessie images to use as a baseline for comparison, and I found that
> useful.
+1

We agreed that we want to use fai for the official Debian images from
stretch. But the users may want to use the new tool for their
customized jessie images, or give it a try before they have to use it
for stretch anyway.

IMO it was a good idea to start with jessie and we should keep this
config in the official git repository.
-- 
regards Thomas

Re: vmdebootstrap plans

[Thomas Lange]

Hi Lars,

I'm really wondering why you want to use YAML for this? Your examples
look like things you could easily implement in shell, if you provide
some functions, subroutines or external programs that implement your
steps. Then your DSL would just be a shell script that the user has to
write.


create_raw_file $size $image
create_partition_table msdos
create_partition $bootsize ext2 /boot bootable
create_partition rest ext4 /
mount_all

and so on.

>   The "step:" key specifies the step to execute. The other keys are
>   parameters for the step. The {{...}} parts are where Jinja2 is used
>   to expand variable values.
These things are very similar to calling commands with parameters in a
shell script without the need of YAML and Jinja2. Much easier and more
common for sysadmins IMO.

Just my two cents.
Thomas

> On Fri, 11 Nov 2016 14:09:39 +, Lars Wirzenius  said:

> * The YAML might look somethin like this:

> - step: create_raw_file
>   size: {{ size }}
>   filename: {{ image }}
> - step: create_partition_table
>   partition_table_type: msdos
> - step: create_partition
>   size: {{ bootsize }}
>   partition_type: ext2
>   bootable: yes
>   mount_point: /boot
> - step: create_partition
>   size: rest
>   partition_type: ext4
>   mount_point: /
> - step: mount_all
> - step: debootstrap
> - step: install_grub
> - step: run_customization_script
>   script: {{ customize }}
> - step: cleanup
> - step: convert_image_to_output_format
>   output_format: qcow2

Re: my progress

[Noah Meyerhans]

On Sun, Nov 13, 2016 at 06:17:33AM -0500, Sam Hartman wrote:
> Noah> I've modified the class/DEBIAN.var file such that the default
> Noah> behavior is to generate images for jessie. We can add a
> Noah> STRETCH class in order to generate images for testing. I'd
> Noah> rather use "stable", but it's preferable to use the release
> Noah> codename in many places and there's no easy way to resolve
> Noah> stable to the current codename AFAIK. Maybe there should be?
> 
> I thought we agreed the fai efforts would be for stretch and we would
> not be converting our existing jessie work.
> I don't mind having jessie support if it's easy, but for example if we
> run into situations where  it adds extra code to work around bugs fixed
> in stretch but present in jessie, I'd like to push back.

Yes, I think you're right.

I based my work on jessie initially because we have JEB's high quality
jessie images to use as a baseline for comparison, and I found that
useful.

In practice, activating a JESSIE or STRETCH class still needs to happen
via the fai-diskimage commandline anyway, and these classes override the
release variable, so defining a default value for that variable in
class/DEBIAN.var was not actually useful. My inclination is to require
the user to activate a specific release class, and for FAI to abort if
one isn't provided.

noah




signature.asc
Description: PGP signature

Re: my progress

[Sam Hartman]

> "Noah" == Noah Meyerhans  writes:

Noah> I've modified the class/DEBIAN.var file such that the default
Noah> behavior is to generate images for jessie. We can add a
Noah> STRETCH class in order to generate images for testing. I'd
Noah> rather use "stable", but it's preferable to use the release
Noah> codename in many places and there's no easy way to resolve
Noah> stable to the current codename AFAIK. Maybe there should be?

I thought we agreed the fai efforts would be for stretch and we would
not be converting our existing jessie work.
I don't mind having jessie support if it's easy, but for example if we
run into situations where  it adds extra code to work around bugs fixed
in stretch but present in jessie, I'd like to push back.

I suspect that long-term find it easier to maintain something where we
have the work-arounds for things we've found in stable releases in a
class, and the main body of the code is targeted at sid and testing.
Thoughts?

Re: ENA kernel driver for Stretch release on AWS cloud

[Marcin Kulisz]

On 2016-11-13 03:37:17, Ben Hutchings wrote:
> On Tue, 2016-11-08 at 14:24 +, Marcin Kulisz wrote:
> > Hi all,
> > 
> > We need ena kernel driver for AWS cloud images as I'm not sure what kernel
> > version we're going to ship with Stretch and this driver is available with
> > 4.9rc, I'd like to ask (and fill the bug report if need):
> > what kernel version are we going to have in Stretch?
> 
> It will be the next kernel version that has upstream long-term support
> (or 4.10 at the latest).  This is planned to be 4.9, although I think
> that hasn't been finalised yet.

Thank you for the info

> > If it's going to be 4.8.X are you ok with me creating bug requesting 
> > including
> > ena driver with it?
> [...]
> 
> Please do that if we announce that we'll be using 4.8.
> 
> For now, I've enabled this driver for the next upload of 4.9 to
> experimental.

Thx a lot Ben it's going to be a great help.
-- 

|_|0|_|  |
|_|_|0|  "Panta rei" |
|0|0|0|  kuLa    |

gpg --keyserver pgp.mit.edu --recv-keys 0x686930DD58C338B3
3DF1  A4DF  C732  4688  38BC  F121  6869  30DD  58C3  38B3


signature.asc
Description: PGP signature