Re: FAI going to the cloud

[Thomas Lange]

> On Wed, 28 Sep 2016 12:22:48 -0700, Derek Poon  
> said:

> Other common formats include VDI, VMDK, and VHD.[1]  Personally, I would 
make it convert to any recognized extension; any other extension on the 
user-supplied image name (whether .raw, .img, or .whatever) would case the 
result not to be converted.
I've added suppport for those formats to fai-diskimage.

-- 
regards Thomas

Re: FAI going to the cloud

[Derek Poon]

On Sep 27, 2016, at 8:24 PM, Thomas Lange  wrote:
> 
> thanks a lot for all your ideas. fai-diskimage was written as a
> wrapper script for several reasons. First I tried to implement it as a
> new action like 'fai install' but I had some problems with that. The
> wrapper script approach worked much better during the
> development.

As long as you've thought it through, that's fine.


> Which flexibility is missing in the wrapper? All you can do in FAI,
> can also be done when calling fai-diskimage.

At the moment, one example I can think of is creating more than one disk image 
at once, such as an OS image and a data image.


> Do you think .raw is the right extension or is .img better? Which other
> formats should be supported?

Other common formats include VDI, VMDK, and VHD.[1]  Personally, I would make 
it convert to any recognized extension; any other extension on the 
user-supplied image name (whether .raw, .img, or .whatever) would case the 
result not to be converted.

I can also imagine that some users might want the installation process to 
produce the result packaged in an OVA, in which case the output would be an 
entire virtual machine rather than just a disk image.  In my opinion, a hook 
for such local requirements would be preferable to a wrapper around a wrapper 
around fai.

[1]: https://spin.atomicobject.com/2013/06/03/ovf-virtual-machine/

Re: FAI going to the cloud

[Thomas Lange]

Hi Derek,

thanks a lot for all your ideas. fai-diskimage was written as a
wrapper script for several reasons. First I tried to implement it as a
new action like 'fai install' but I had some problems with that. The
wrapper script approach worked much better during the
development. Second, fai-diskimage has some other options, and will get
more diks image specific options in the future. Currently I do not see
much disadvantages in the wrapper script approach.

> The second issue is flexibility.  The fai-diskimage script doesn't really 
do that much: it creates and mounts an image at the beginning, then possibly 
converts the resulting image to QCOW2 format afterwards.  If the fai command 
were tweaked to allow a size to be specified, then the image creation could be 
accomplished with a hook for the partition task (or perhaps a new diskimage 
task).
Which flexibility is missing in the wrapper? All you can do in FAI,
can also be done when calling fai-diskimage.

> Whether the format conversion is needed could be inferred from whether 
the image filename ends in ".qcow2".  The conversion to QCOW2 could be done 
using a hook for faiend.
Yes, I also was thinking about detecting the image format on the name
suffix. Maybe I will change this and remove the -C option. Do you
think .raw is the right extension or is .img better? Which other
formats should be supported?

-- 
regards Thomas

Re: FAI going to the cloud

[Derek Poon]

On Sep 22, 2016, at 5:48 PM, Thomas Lange  wrote:
> 
>> On Thu, 22 Sep 2016 17:31:54 -0700, Derek Poon  
>> said:
> 
>> Instead of fai-diskimage as a separate command, have you considered making 
>> it a mode of the fai command itself, analogous to `fai dirinstall`?  It 
>> would be nice to be able to write hooks for diskimage installs.
> fai-diskimage is a seperate script which calls "fai install"
> internally. You can write hooks for fai-diskinstall the same way as in
> fai itself, because it uses the same config space as fai.


I was actually trying to make two observations.

The first issue is user experience consistency.  We have:

   fai install
   fai sysinfo
   fai dirinstall /path/to/target/directory
   fai-diskimage -S 32G /path/to/image

The last one seems conceptually similar to dirinstall.  Why should it invoked 
differently, as "fai-diskimage" rather than "fai diskimage"?  Furthermore, the 
fai-diskimage command is hard-coded to do the "install" action, so these 
choices are mutually exclusive.

To me, "fai-diskimage" is closer in nature to "fai install" than to "fai-cd" or 
"fai-kvm", and should therefore behave similarly.


The second issue is flexibility.  The fai-diskimage script doesn't really do 
that much: it creates and mounts an image at the beginning, then possibly 
converts the resulting image to QCOW2 format afterwards.  If the fai command 
were tweaked to allow a size to be specified, then the image creation could be 
accomplished with a hook for the partition task (or perhaps a new diskimage 
task).

Whether the format conversion is needed could be inferred from whether the 
image filename ends in ".qcow2".  The conversion to QCOW2 could be done using a 
hook for faiend.

Personally, I would prefer the configuration approach over the wrapper script, 
since it allows for customization better.  (For example, it would be easy to 
add support for image formats other than QCOW2.)


On top of that, there would be one fewer man page to write.

I hope you will give these thoughts some consideration before making the next 
FAI release.

Derek

Re: FAI going to the cloud

[Thomas Lange]

> On Thu, 22 Sep 2016 17:31:54 -0700, Derek Poon  
> said:

> Instead of fai-diskimage as a separate command, have you considered 
making it a mode of the fai command itself, analogous to `fai dirinstall`?  It 
would be nice to be able to write hooks for diskimage installs.
fai-diskimage is a seperate script which calls "fai install"
internally. You can write hooks for fai-diskinstall the same way as in
fai itself, because it uses the same config space as fai.
-- 
regards Thomas

Re: FAI going to the cloud

[Derek Poon]

On Jul 29, 2016, at 7:27 AM, Thomas Lange  wrote:
> 
> A new beta version of FAI is available. It now includes a new command
> for creating cloud^Wdisk images. fai-diskimage can create a raw or
> qcow2 disk image without starting a virtual machine. This image is
> ready-to-boot for your VM or cloud instance. Currently it has no man
> page, but a help is printed when calling with -h.
> 
> I plan to release FAI 5.2 in september, so any feedback is welcome.
> 
> Here's a log from a call of fai-diskimage:
> 
> # fai-diskimage -u cloudhost -S900M 
> -cDEFAULT,DEBIAN,AMD64,FAIBASE,DEMO,GRUB_PC,CLOUD /tmp/disk

Thomas,

Instead of fai-diskimage as a separate command, have you considered making it a 
mode of the fai command itself, analogous to `fai dirinstall`?  It would be 
nice to be able to write hooks for diskimage installs.

Derek Poon
University of British Columbia, IT Services

Re: FAI going to the cloud

[Andrew Ruthven]

On Fri, 2016-07-29 at 16:27 +0200, Thomas Lange wrote:
> A new beta version of FAI is available. It now includes a new command
> for creating cloud^Wdisk images. fai-diskimage can create a raw or
> qcow2 disk image without starting a virtual machine. This image is
> ready-to-boot for your VM or cloud instance. Currently it has no man
> page, but a help is printed when calling with -h.

Oh, this looks interesting. I was planning on doing something similar
to how we've been using FAI to build Linux-VServers. Use a standard
image, then copy up the FAI config directory and for a fai-softupdate.

> I plan to release FAI 5.2 in september, so any feedback is welcome.

I'll give it a whirl in the new week.

Cheers,
Andrew
-- 
Andrew Ruthven, Wellington, New Zealand
and...@etc.gen.nz | linux.conf.au 2016 
  New Zealand's only Cloud:   | LCA By the Bay, Geelong, AU
https://catalyst.net.nz/cloud | http://lca2016.linux.org.au

FAI going to the cloud

[Thomas Lange]

A new beta version of FAI is available. It now includes a new command
for creating cloud^Wdisk images. fai-diskimage can create a raw or
qcow2 disk image without starting a virtual machine. This image is
ready-to-boot for your VM or cloud instance. Currently it has no man
page, but a help is printed when calling with -h.

I plan to release FAI 5.2 in september, so any feedback is welcome.

Here's a log from a call of fai-diskimage:

# fai-diskimage -u cloudhost -S900M 
-cDEFAULT,DEBIAN,AMD64,FAIBASE,DEMO,GRUB_PC,CLOUD /tmp/disk
Formatting '/tmp/disk.raw', fmt=raw size=943718400 
 -
   Fully Automatic Installation  -  FAI

   5.2~beta11   (c) 1999-2016
   Thomas Lange  
 -
Starting FAI execution - 20160729_153848
Hostname set to cloudhost

Using configuration files from /etc/fai
Calling task_confdir
FAI_FLAGS: 
Setting SERVER=. Value extracted from FAI_CONFIG_SRC.
No monitor daemon defined.
FAI_CONFIG_SRC is set to file:///srv/fai/config
Using local path /srv/fai/config as config space.
Source hook: setup.DEFAULT.sh
setup.DEFAULT.sh OK.
Calling task_setup
FAI_FLAGS: 
Calling task_defclass
List of all classes:  DEFAULT DEBIAN AMD64 FAIBASE DEMO GRUB_PC CLOUD
Calling task_defvar
++ CONSOLEFONT=
++ KEYMAP=us-latin1
++ MODULESLIST='usbhid psmouse'
++ FAI_RAMDISKS='/tmp/fai-diskimage.Di5gWk/var/lib/dpkg 
/tmp/fai-diskimage.Di5gWk/var/cache'
++ FAI_ALLOW_UNSIGNED=1
++ UTC=yes
++ TIMEZONE=Europe/Berlin
++ ROOTPW=X
++ STOP_ON_ERROR=700
++ MAXPACKAGES=800
Calling task_action
FAI_ACTION: install
Performing FAI installation. All data may be overwritten!
Calling task_install
Calling task_partition
Starting setup-storage 2.0
Using config file: /srv/fai/config/disk_config/CLOUD
Parted could not read a disk label (new disk?)
Executing: parted -s /dev/loop0 mklabel msdos
DEGRADED MODE. Incomplete RAID LVs will be processed.
Finding all volume groups
  No volume groups found
Executing: parted -s /dev/loop0 mklabel msdos
Executing: parted -s /dev/loop0 mkpart primary "ext3" 1048576B 854589439B
Executing: parted -s /dev/loop0 set 1 boot on
Executing: parted -s /dev/loop0 mkpart primary "ext3" 854589440B 943718399B
Executing: mkfs.ext4  /dev/loop0p1
Executing: tune2fs -c 0 -i 0 /dev/loop0p1
Executing: mkfs.ext4 -L home -m 1 /dev/loop0p2
Executing: tune2fs -c 0 -i 0 /dev/loop0p2
/dev/loop0p2 UUID=300fd2ca-7363-4b74-a41d-969dd722894b
/dev/loop0p2 LABEL=home
/dev/loop0p1 UUID=138cf8b3-bb1d-426b-bdfc-7e27fbedcf2a
Calling task_mountdisks
Calling task_extrbase
Unpacking distro base archive
ftar: extracting /srv/fai/config/basefiles/CLOUD.tar.xz to 
/tmp/fai-diskimage.Di5gWk/
Calling task_debconf
Calling task_repository
'/etc/resolv.conf' -> '/tmp/fai-diskimage.Di5gWk/etc/resolv.conf'
'/etc/hosts' -> '/tmp/fai-diskimage.Di5gWk/etc/hosts' (backup: 
'/tmp/fai-diskimage.Di5gWk/etc/hosts.orig')
fcopy: no matching file for any class for etc/apt/sources.list defined.
'/etc/fai/apt/sources.list' -> '/tmp/fai-diskimage.Di5gWk/etc/apt/sources.list'
fcopy: no matching file for any class for etc/apt/sources.list defined.
fcopy: copied /srv/fai/config/files/etc/apt/apt.conf.d/force_confdef/DEBIAN to 
/tmp/fai-diskimage.Di5gWk/etc/apt/apt.conf.d/force_confdef
Loading APT key from DEBIAN.asc OK
Calling hook: updatebase.DEBIAN
updatebase.DEBIANOK.
Calling task_updatebase
Calling hook: instsoft.DEBIAN
fcopy: destination etc/apt/apt.conf.d/force_confdef remains unchanged
instsoft.DEBIAN  OK.
Calling task_instsoft
Installing software may take a while
Calling task_configure
Executingshell: DEBIAN/10-rootpw
DEBIAN/10-rootpw OK.
Executingshell: DEBIAN/20-capabilities
DEBIAN/20-capabilities OK.
Executingshell: DEBIAN/30-interface
DEBIAN/30-interface  OK.
Executingshell: DEBIAN/40-misc
DEBIAN/40-misc   OK.
Executingshell: FAIBASE/10-misc
FAIBASE/10-misc  OK.
Executingshell: FAIBASE/20-removable_media
FAIBASE/20-removable_media OK.
Executingshell: DEMO/10-misc
DEMO/10-misc OK.
Executingshell: GRUB_PC/10-setup
GRUB_PC/10-setup OK.
Executingshell: CLOUD/99-cleanup
CLOUD/99-cleanup OK.
Calling task_tests
Calling task_finish
Filesystem  Type  Size  Used Avail Use% Mounted on
/dev/loop0p1ext4  786M  627M  103M  86% /tmp/fai-diskimage.Di5gWk
/dev/loop0p2ext4   79M  1.6M   75M   3% 
/tmp/fai-diskimage.Di5gWk/home
Fri Jul 29 15:39:29 CEST 2016
The install took 41 seconds.
Calling task_chboot
Calling task_savelog
$LOGUSER is undefined. Not saving log files to remote.
Calling task_faiend
End of /usr/sbin/fai
Size of disk image and filename: 649M   /tmp/disk.raw
Image created by: /usr/sbin/fai-diskimage -u cloudhost -S900M 
-cDEFAULT,DEBIAN,AMD64,FAIBASE,DEMO,GRUB_PC,CLOUD /tmp/disk
-- 
regards Thomas (Mrfai)