Bug#863388: live-boot: Added new functionality to live-boot: rootcopy, modularized filesystems, package installer

[Raphael Hertzog]

Hello guys,

I tried to merged all pending patches recently and I just came on
this discussion and this bug submission.

On Fri, 16 Jun 2017, William Prochazka wrote:
> I concede that I didn't realize the module file could include a raw
> directory since the man page refers to disk images.

Maybe you can send a patch to update the documentation too?

On Tue, 20 Jun 2017, Mert Dirik wrote:
> I should note that I'm just an outsider of this project so I don't have a
> say in what can or can not be accepted. I only replied because added
> functionality seemed redundant and I wanted to point it out. Now that I got
> my message across I count myself successfull :). I'm aware there is also
> some other functionality in your patch but I didn't really checked them and
> you should decide whether to clean them up to leverage existing
> functionality and push it for inclusion.

Thank you for your comments anyway. It helped to put things into
perspective.

William, I would like you to send me an update on all this and to resubmit
the patches which are still relevant as merge request on
https://salsa.debian.org/live-team/live-boot

Please don't mix multiple features in a single commit. It's easier to
review when clearly separated. And if they don't depend on each other,
then submit them as different merge requests, it's even better.

Please include updated documentation in your patch. This was clearly
missing from what I saw quickly.

Note that I pushed multiple changes recently (and notably the switch
to /run/live instead of /lib/live/mount) which will impact your patch.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Bug#863388: live-boot: Added new functionality to live-boot: rootcopy, modularized filesystems, package installer

[Mert Dirik]

On 06/16/2017 05:13 AM, William Prochazka wrote:

Ahhh, a thousand apologies as my original submission didn't make it
clear.  The module stuff is all related to using the toram function.
When using toram, the module function doesn't work as expected.

If you specify "toram=something.module" (not documented in man page, but
the arg is parsed into MODULETORAM), it assumes this is a single
squashfs filesystem, not a module file (maybe just confusion with what a
"module" is).
If you specify "module=foo toram", it ignores any foo.module and tries
to copy the whole live directory.  Maybe then it honors the module file,
but I never get that far as I run out of RAM ;)

I can clean up the patch but the additions to the toram file check to
see if:
a) module = something and  toram, then calculate size based on the sum
of the parts in the module file and only copy those parts to ram, then
continue as before.

The normal behavior for toram=filesystem.squashfs is kept and toram
without a "module=" is also kept as before...really just takes care of
loading a module into ram.



Now it is clear, I'm sorry I missed it was about toram functionality, 
thanks for pointing out.


I haven't studied details of toram code but I used it enough to know it 
is somewhat inflexible (my pain point was trying to use live-toram tool 
with a non-squashfs image). Extending and polishing it would be appreciated.





I concede that I didn't realize the module file could include a raw
directory since the man page refers to disk images.  It is a bit of a
pain to make a module file for each bootable filesystem you may want,
but I can make due with that if you don't like the actual rootcopy
cheatcode.  I might also point out that I would bet many people are


I should note that I'm just an outsider of this project so I don't have 
a say in what can or can not be accepted. I only replied because added 
functionality seemed redundant and I wanted to point it out. Now that I 
got my message across I count myself successfull :). I'm aware there is 
also some other functionality in your patch but I didn't really checked 
them and you should decide whether to clean them up to leverage existing 
functionality and push it for inclusion.




unaware of this feature since it is buried in the module functions and
not well explained in the man page, but thanks for clearing that up.


It is indeed an important issue with live-boot documentation. It is a 
very capable set of scripts but so many times I've had to dig up the 
code to see how to use a functionality or whether a specific use case in 
my mind was actually supported or not. It is often possible to rely on 
implementation details to use it in ways it was originally thought or at 
least publicly documented.





On Fri, 16 Jun 2017 00:47:56 +0300 Mert Dirik > wrote:


On Wed, 14 Jun 2017 01:48:05 + William Prochazka wrote:
 > Well,
 >
 > 1. Root copy is not a squashfs filesystem, so it allows for quick
tweaking
 > and modification without having to "commit" your changes. This also

makes

 > modifications (for the purpose of troubleshooting) much easier.
 >
 > 2. The alternative filesystem module is still a single file. The new
 > feature allow for the specification and ordering of filesystem modules.
 > For example, imagine the following:
 > filesyste A contains all necessary files to boot a system into a basic
 > shell with basic functionality, nothing more. Branch filesystem B
contains
 > a bunch of advanced tools (coimpilers, scripting environments, etc.
Branch
 > filesystem C contains X and Branch Filesystem D contains all of the X
Bloat
 > applications.
 > Together, all four branches are roughly the size that the one
"monolithic"
 > filesystem D would have been if I wanted to use the legacy modules
 > directive, but now, I have a boot time option to specify if I want

just A

 > for a tiny system, (A and B) for more functionality, (A, B, and C)

for a

 > GUI, or (A, B, C, and D) for all of the bells and whistle. Now I

get all

 > of that and don't have four copies of the filesystem. Furthermore, if I
 > then add a file to A, It will be present when booting into D due to
 > squashfs.
 >
 > It is really nice.

I'm still not getting it. You can accomplish all of these with the
existing module functionality.

Modules don't have to be squashfs files. They can also be regular
directories. You can modify them freely. You can use them in the exact
same way you use overlay directories, because they end up being just
another layer of  the overlayfs mount. You can mix and match modules
using *.module files or change their ordering, because they are just
overlays. You can also select different *.module files with boot

parameters.


Sorry if I still miss something obvious but this seems to cover all
functionality you've brought so far.

Bug#863388: live-boot: Added new functionality to live-boot: rootcopy, modularized filesystems, package installer

[William Prochazka]

Ahhh, a thousand apologies as my original submission didn't make it clear.
The module stuff is all related to using the toram function.  When using
toram, the module function doesn't work as expected.

If you specify "toram=something.module" (not documented in man page, but
the arg is parsed into MODULETORAM), it assumes this is a single squashfs
filesystem, not a module file (maybe just confusion with what a "module"
is).
If you specify "module=foo toram", it ignores any foo.module and tries to
copy the whole live directory.  Maybe then it honors the module file, but I
never get that far as I run out of RAM ;)

I can clean up the patch but the additions to the toram file check to see
if:
a) module = something and  toram, then calculate size based on the sum of
the parts in the module file and only copy those parts to ram, then
continue as before.

The normal behavior for toram=filesystem.squashfs is kept and toram without
a "module=" is also kept as before...really just takes care of loading a
module into ram.

I concede that I didn't realize the module file could include a raw
directory since the man page refers to disk images.  It is a bit of a pain
to make a module file for each bootable filesystem you may want, but I can
make due with that if you don't like the actual rootcopy cheatcode.  I
might also point out that I would bet many people are unaware of this
feature since it is buried in the module functions and not well explained
in the man page, but thanks for clearing that up.

On Fri, 16 Jun 2017 00:47:56 +0300 Mert Dirik  wrote:
>
> On Wed, 14 Jun 2017 01:48:05 + William Prochazka wrote:
>  > Well,
>  >
>  > 1. Root copy is not a squashfs filesystem, so it allows for quick
> tweaking
>  > and modification without having to "commit" your changes. This also
makes
>  > modifications (for the purpose of troubleshooting) much easier.
>  >
>  > 2. The alternative filesystem module is still a single file. The new
>  > feature allow for the specification and ordering of filesystem modules.
>  > For example, imagine the following:
>  > filesyste A contains all necessary files to boot a system into a basic
>  > shell with basic functionality, nothing more. Branch filesystem B
> contains
>  > a bunch of advanced tools (coimpilers, scripting environments, etc.
> Branch
>  > filesystem C contains X and Branch Filesystem D contains all of the X
> Bloat
>  > applications.
>  > Together, all four branches are roughly the size that the one
> "monolithic"
>  > filesystem D would have been if I wanted to use the legacy modules
>  > directive, but now, I have a boot time option to specify if I want
just A
>  > for a tiny system, (A and B) for more functionality, (A, B, and C) for
a
>  > GUI, or (A, B, C, and D) for all of the bells and whistle. Now I get
all
>  > of that and don't have four copies of the filesystem. Furthermore, if I
>  > then add a file to A, It will be present when booting into D due to
>  > squashfs.
>  >
>  > It is really nice.
>
> I'm still not getting it. You can accomplish all of these with the
> existing module functionality.
>
> Modules don't have to be squashfs files. They can also be regular
> directories. You can modify them freely. You can use them in the exact
> same way you use overlay directories, because they end up being just
> another layer of  the overlayfs mount. You can mix and match modules
> using *.module files or change their ordering, because they are just
> overlays. You can also select different *.module files with boot
parameters.
>
> Sorry if I still miss something obvious but this seems to cover all
> functionality you've brought so far.
>
>

Bug#863388: live-boot: Added new functionality to live-boot: rootcopy, modularized filesystems, package installer

[Mert Dirik]

On Wed, 14 Jun 2017 01:48:05 + William Prochazka wrote:
> Well,
>
> 1. Root copy is not a squashfs filesystem, so it allows for quick 
tweaking

> and modification without having to "commit" your changes. This also makes
> modifications (for the purpose of troubleshooting) much easier.
>
> 2. The alternative filesystem module is still a single file. The new
> feature allow for the specification and ordering of filesystem modules.
> For example, imagine the following:
> filesyste A contains all necessary files to boot a system into a basic
> shell with basic functionality, nothing more. Branch filesystem B 
contains
> a bunch of advanced tools (coimpilers, scripting environments, etc. 
Branch
> filesystem C contains X and Branch Filesystem D contains all of the X 
Bloat

> applications.
> Together, all four branches are roughly the size that the one 
"monolithic"

> filesystem D would have been if I wanted to use the legacy modules
> directive, but now, I have a boot time option to specify if I want just A
> for a tiny system, (A and B) for more functionality, (A, B, and C) for a
> GUI, or (A, B, C, and D) for all of the bells and whistle. Now I get all
> of that and don't have four copies of the filesystem. Furthermore, if I
> then add a file to A, It will be present when booting into D due to
> squashfs.
>
> It is really nice.

I'm still not getting it. You can accomplish all of these with the 
existing module functionality.


Modules don't have to be squashfs files. They can also be regular 
directories. You can modify them freely. You can use them in the exact 
same way you use overlay directories, because they end up being just 
another layer of  the overlayfs mount. You can mix and match modules 
using *.module files or change their ordering, because they are just 
overlays. You can also select different *.module files with boot parameters.


Sorry if I still miss something obvious but this seems to cover all 
functionality you've brought so far.

Bug#863388: live-boot: Added new functionality to live-boot: rootcopy, modularized filesystems, package installer

[William Prochazka]

Well,

1. Root copy is not a squashfs filesystem, so it allows for quick tweaking
and modification without having to "commit" your changes.  This also makes
modifications (for the purpose of troubleshooting) much easier.

2. The alternative filesystem module is still a single file.  The new
feature allow for the specification and ordering of filesystem modules.
For example, imagine the following:
filesyste A contains all necessary files to boot a system into a basic
shell with basic functionality, nothing more.  Branch filesystem B contains
a bunch of advanced tools (coimpilers, scripting environments, etc.  Branch
filesystem C contains X and Branch Filesystem D contains all of the X Bloat
applications.
Together, all four branches are roughly the size that the one "monolithic"
filesystem D would have been if I wanted to use the legacy modules
directive, but now, I have a boot time option to specify if I want just A
for a tiny system, (A and B) for more functionality, (A, B, and C) for a
GUI, or (A, B, C, and D) for all of the bells and whistle.  Now I get all
of that and don't have four copies of the filesystem.  Furthermore, if I
then add a file to A, It will be present when booting into D due to
squashfs.

It is really nice.

Bug#863388: live-boot: Added new functionality to live-boot: rootcopy, modularized filesystems, package installer

[Mert Dirik]

On 05/26/2017 07:04 AM, Bill Prochazka wrote:


1. rootcopy:  This allows for one or more folders on the boot medium that
contain a folder structure relative to /.  before init, these files are copied
to the rootfs.  This feature allows users to make small changes to their
environment (like wallpaper changes or dekstop icons) without needing to
remaster or have a persistent volume.

2. modularized filesystems: This allows for the use of a file, *.module, that
contains newlibe seperated squashfs filenames to be loaded in order.  This is a
change from the default behavior of coping all squashfs files and mounting them
in alphabetical order.



What does those two points help to accomplish that cannot be done with 
the existing "module=..." argument?



   module=NAME
   Instead of using the default optional file "filesystem.module" (see below) 
another file could be specified without the extension ".module"; it should be
   placed on "/live" directory of the live medium.


Using this argument you can create overlay directory hierarchies and/or 
squashfs images and mount them in an arbitrary order.

Bug#863388: live-boot: Added new functionality to live-boot: rootcopy, modularized filesystems, package installer

[Bill Prochazka]

Package: live-boot
Version: 4.0.2-1
Severity: wishlist
Tags: patch

Dear Maintainer,
There are a few features I have come to love in other live systems that are not
available in the current live-boot system.  I have attached a patch that adds
these features and think it maybe valuable upstream.
1. rootcopy:  This allows for one or more folders on the boot medium that
contain a folder structure relative to /.  before init, these files are copied
to the rootfs.  This feature allows users to make small changes to their
environment (like wallpaper changes or dekstop icons) without needing to
remaster or have a persistent volume.

2. modularized filesystems: This allows for the use of a file, *.module, that
contains newlibe seperated squashfs filenames to be loaded in order.  This is a
change from the default behavior of coping all squashfs files and mounting them
in alphabetical order.

3. packge installer: This allows for a user to include a small apt repo in the
live media.  By creating a *.package file, a user can tell the system to
install these packages from the repo on boot before init.  This is helpful for
testing packages without having to remaster.

I have been using these for quite some time and found them to be critical in my
use.  I would like inclusion upstream so I don't have to continually patch as
new versions are released.



-- Package-specific info:

-- System Information:
Debian Release: 8.7
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages live-boot depends on:
ii  live-boot-initramfs-tools [live-boot-backend]  4.0.2-1

Versions of packages live-boot recommends:
pn  live-boot-doc  
pn  live-tools 
ii  rsync  3.1.1-3
pn  uuid-runtime   

Versions of packages live-boot suggests:
ii  cryptsetup2:1.7.3-3
pn  curlftpfs 
pn  httpfs2   
pn  unionfs-fuse  
ii  wget  1.16-1+deb8u1

-- debconf information:
Unescaped left brace in regex is deprecated, passed through in regex; marked by 
<-- HERE in m/^(.*?)(\\)?\${ <-- HERE ([^{}]+)}(.*)$/ at 
/usr/share/perl5/Debconf/Question.pm line 72.
Unescaped left brace in regex is deprecated, passed through in regex; marked by 
<-- HERE in m/\${ <-- HERE ([^}]+)}/ at /usr/share/perl5/Debconf/Config.pm line 
30.
diff -ruN lib/live/boot/9990-cleanup.sh /lib/live/boot/9990-cleanup.sh
--- lib/live/boot/9990-cleanup.sh	1970-01-01 00:00:00.0 +
+++ /lib/live/boot/9990-cleanup.sh	2017-05-15 15:59:13.0 +
@@ -0,0 +1,24 @@
+#!/bin/sh
+
+#set -e
+
+
+
+Cleanup ()
+{
+	. /bin/live-boot
+	. /scripts/functions
+	log_begin_msg "Cleaning up your mess..."
+	# Unmount boot media if we did a rootcopy or module install
+	([ -n ${ROOTCOPY} ] || [ -n ${MODULES} ]) && umount /root/lib/live/mount/medium
+	#[ -n ${MODULES} ] && umount /root/lib/live/mount/medium
+
+	# Copy through initrd versions of live-boot scripts
+	cp -af /lib/live/boot/* /root/lib/live/boot/
+	cp -af /scripts/live /root/usr/share/initramfs-tools/scripts/live
+
+	# Remove duplicate /live/overlay
+	umount /root/lib/live/mount/overlay
+maybe_break cleanup	
+	log_end_msg
+}
diff -ruN lib/live/boot/9990-cmdline-old /lib/live/boot/9990-cmdline-old
--- lib/live/boot/9990-cmdline-old	2014-12-10 09:37:04.0 +
+++ /lib/live/boot/9990-cmdline-old	2017-05-26 03:45:29.229477053 +
@@ -208,7 +208,8 @@
 
 			toram)
 TORAM="true"
-export TORAM
+MODULETORAM="filesystem.squashfs"
+export TORAM MODULETORAM
 ;;
 
 			toram=*)
@@ -241,6 +242,31 @@
 UNIONTYPE="${_PARAMETER#union=}"
 export UNIONTYPE
 ;;
+
+			rootcopy)
+ROOTCOPY="true"
+export ROOTCOPY
+;;
+
+			rootcopy=*)
+ROOTCOPY="${_PARAMETER#rootcopy=}"
+export ROOTCOPY
+;;
+
+			packagesdir=*)
+PACKAGESDIR="${_PARAMETER#packagesdir=}"
+export PACKAGESDIR
+;;
+
+			packages=*)
+PACKAGES="${_PARAMETER#packages=}"
+export PACKAGES
+;;
+
+			repodir=*)
+REPODIR="${_PARAMETER#repodir=}"
+export REPODIR
+;;
 		esac
 	done
 
diff -ruN lib/live/boot/9990-main.sh /lib/live/boot/9990-main.sh
--- lib/live/boot/9990-main.sh	2014-12-10 09:37:04.0 +
+++ /lib/live/boot/9990-main.sh	2017-05-26 03:12:30.501381288 +
@@ -211,4 +211,17 @@
 	exec 2>&7 7>&-
 	kill ${tailpid}
 	[ -w "${rootmnt}/var/log/" ] && mkdir -p "${rootmnt}/var/log/live" && cp boot.log "${rootmnt}/var/log/live" 2>/dev/null
+
+	# install user modules
+	Packages
+
+	# Clone Root mods
+	Rootcopy
+
+	# Cleanup
+	Cleanup
+
+	# Copy through initrd versions of live-boot scripts
+	cp -af /lib/live/boot/* /root/lib/live/boot/
+	cp -af /scripts/live /root/usr/share/initramfs-tools/scripts/live
 }
diff -ruN lib/live/boot/9990-misc-helpers.sh /lib/live/boot/9990-misc-helpers.sh
---