hi list, I want to open a discussion on how to pimp the great live-helper scripts.
as startpoint I have attached a patch against1.0~a41+20080325.095215 which
does the following:
1. while doing updates for the cache, instead of deleting a cache & copying it
complete over, it checks whether rsync is installed and does an
rsync --delete -a source/ target/
advantage:
assuming that the most packages won't change between two builds,
we only need to copy the updated files. leftovers are deleted.
extrapackages needed:
rsync
2. while we using aufs only within debianLive, here aufs is used to mount the
chroot/chroot construction instead of doubling all to chroot.tmp and move it
to chroot/chroot.
this is done by renaming chroot to .chroot,
creating a chroot.binary and chroot.tmp/...
and mount our source .chroot (readonly) to chroot.binary and to
chroot.binary/chroot, each with a rw-aufs-layer.
chroot.binary is then sym-linked to chroot, to better recognize that the
aufs-mechanic is in use.
this gives us
.chroot: our original untouched chroot, as lh_chroot created it.
chroot.tmp: the aufs cow-directory, where any changes appear for chroot.binary
chroot.binary: mounted .chroot(ro) with aufs-overlay chroot.tmp(rw)
chroot.tmp/.chroot.tmp: the aufs cow-directory, where any changes appear
for chroot.binary/chroot
chroot.binary/chroot: mounted .chroot(ro) with aufs-overlay
chroot.tmp/.chroot.tmp(rw)
chroot -> chroot.binary: a sym-link for better recognizing.
advantages:
* the whole construction of chroot/chroot is done in a few seconds,
instead of doubling MB/GB of data within 10th of
minutes (your mileage my vary)
* only space for one chroot dir. is needed, instead of double
extrapackages needed:
aufs-modules & aufs-tools
this patch is a hack and is meant to start a discussion, even so it worked for
me for a while perfectly, building usb-images. before it does anything new,
it checks on existence of rsync or aufs and aufs-tools and is reentrant for
lh_binary, lh_binary_chroot.
lh_binary_rootfs is turning it back to normal, removing all tmp. files/dirs.
also attached is a little script, which can be sourced to get some
shellfunctions which come in handy, while playing around with the
aufs-construct:
lh_aufs [-i] [-u]: to mount & umount the aufs construction
lh_mount: to mount proc, devpts & sysfs:
lh_umount: to umount proc, devpts & sysfs:
KUDOS to Daniel & the developer gang for providing such a great tool.
guenter
diff -rud helpers.1.0~a41+20080325.095215/lh_binary_chroot helpers.mod/lh_binary_chroot
--- helpers.1.0~a41+20080325.095215/lh_binary_chroot 2008-03-25 20:18:22.400052500 +0100
+++ helpers.mod/lh_binary_chroot 2008-03-25 21:40:31.463049491 +0100
@@ -47,6 +47,11 @@
# Creating lock file
Create_lockfile .lock
+#FIXME
+${LH_ROOT_COMMAND} umount -f chroot/sys > /dev/null 2>&1 || true
+${LH_ROOT_COMMAND} umount -f chroot/proc > /dev/null 2>&1 || true
+${LH_ROOT_COMMAND} umount -f chroot/dev/pts > /dev/null 2>&1 || true
+
# Normally, virtual filesystems are not mounted here, but people tend to be lazy
if [ -f chroot/proc/version ]
then
@@ -70,6 +75,17 @@
fi
fi
+if [ `grep -c devpts-live /proc/mounts` -gt 0 ]
+then
+ if [ "${LH_USE_FAKEROOT}" != "enabled" ]
+ then
+ ${LH_ROOT_COMMAND} umount chroot/dev/pts
+ else
+ rm -rf chroot/sys
+ mkdir -p chroot/sys
+ fi
+fi
+
# Copying /dev if using fakeroot
if [ "${LH_USE_FAKEROOT}" = "enabled" ]
then
@@ -90,12 +106,39 @@
Echo_message "This may take a while."
# Removing old chroot
-${LH_ROOT_COMMAND} rm -rf chroot/chroot
-${LH_ROOT_COMMAND} rm -rf chroot.tmp
+if [ -d .chroot ];
+then
+ # kill process, which are started during installation and left running
+ kill -15 $(lsof +D chroot -F p|cut -c 2-)
-# Copying new chroot
-${LH_ROOT_COMMAND} cp -a chroot chroot.tmp
-${LH_ROOT_COMMAND} mv chroot.tmp chroot/chroot
+ ${LH_ROOT_COMMAND} umount chroot.binary/chroot chroot.binary
+ ${LH_ROOT_COMMAND} rm -rf chroot.binary chroot.tmp
+ ${LH_ROOT_COMMAND} rm -rf chroot
+ ${LH_ROOT_COMMAND} mv .chroot chroot
+else
+ ${LH_ROOT_COMMAND} rm -rf chroot/chroot
+ ${LH_ROOT_COMMAND} rm -rf chroot.tmp
+fi
+
+# to speed up dramatically, use aufs-mounted chroot/chroot instead of copying
+
+# load aufs - be true even when compiled statically into kernel (-> i.e. necessary for Realtime-Patch to work)
+modprobe -q aufs || true
+
+#check on availability of tools & aufs
+if [ -x /usr/sbin/mount.aufs -a `grep -c aufs /proc/filesystems` -gt 0 ];
+then
+ ${LH_ROOT_COMMAND} mv chroot .chroot
+ ${LH_ROOT_COMMAND} mkdir -p chroot.binary chroot.tmp/{chroot,.chroot.tmp}
+ ${LH_ROOT_COMMAND} mount none chroot.binary/ -t aufs -o br:chroot.tmp:.chroot=ro+wh
+ # we need the chroot/chroot with whiteout cap. to use config/binary_rootfs/exclude
+ ${LH_ROOT_COMMAND} mount none chroot.binary/chroot -t aufs -o br:chroot.tmp/.chroot.tmp:.chroot=ro+wh
+ ${LH_ROOT_COMMAND} ln -s chroot.binary chroot
+else
+ # Copying new chroot
+ ${LH_ROOT_COMMAND} cp -a chroot chroot.tmp
+ ${LH_ROOT_COMMAND} mv chroot.tmp chroot/chroot
+fi
if [ -f config/binary_rootfs/exclude ]
then
diff -rud helpers.1.0~a41+20080325.095215/lh_binary_rootfs helpers.mod/lh_binary_rootfs
--- helpers.1.0~a41+20080325.095215/lh_binary_rootfs 2008-03-25 20:18:22.420033660 +0100
+++ helpers.mod/lh_binary_rootfs 2008-03-25 16:01:40.519049891 +0100
@@ -125,7 +125,25 @@
# Move image
mv chroot/filesystem.${LH_CHROOT_FILESYSTEM} binary/${INITFS}
- rm -rf chroot/chroot
+ if [ -d .chroot ]; then
+ # kill process, which are started during installation and left running
+ kill -15 $(lsof +D chroot -F p|cut -c 2-)
+
+ ${LH_ROOT_COMMAND} umount -f chroot/sys > /dev/null 2>&1 || true
+ ${LH_ROOT_COMMAND} umount -f chroot/proc > /dev/null 2>&1 || true
+ ${LH_ROOT_COMMAND} umount -f chroot/dev/pts > /dev/null 2>&1 || true
+
+ ${LH_ROOT_COMMAND} umount -f chroot.binary/chroot chroot.binary > /dev/null 2>&1 || true
+ ${LH_ROOT_COMMAND} rm -rf chroot.binary chroot.tmp
+ ${LH_ROOT_COMMAND} rm -rf chroot
+ ${LH_ROOT_COMMAND} mv .chroot chroot
+
+ ${LH_ROOT_COMMAND} mount devpts-live -t devpts chroot/dev/pts || true
+ ${LH_ROOT_COMMAND} mount proc-live -t proc chroot/proc
+ ${LH_ROOT_COMMAND} mount sysfs-live -t sysfs chroot/sys
+ else
+ ${LH_ROOT_COMMAND} rm -rf chroot/chroot
+ fi
;;
disabled)
@@ -167,7 +185,25 @@
# Move image
mv chroot/filesystem.jffs2 binary/${INITFS}
- rm -rf chroot/chroot
+ if [ -d .chroot ]; then
+ # kill process, which are started during installation and left running
+ kill -15 $(lsof +D chroot -F p|cut -c 2-)
+
+ ${LH_ROOT_COMMAND} umount -f chroot/sys > /dev/null 2>&1 || true
+ ${LH_ROOT_COMMAND} umount -f chroot/proc > /dev/null 2>&1 || true
+ ${LH_ROOT_COMMAND} umount -f chroot/dev/pts > /dev/null 2>&1 || true
+
+ ${LH_ROOT_COMMAND} umount -f chroot.binary/chroot chroot.binary > /dev/null 2>&1 || true
+ ${LH_ROOT_COMMAND} rm -rf chroot.binary chroot.tmp
+ ${LH_ROOT_COMMAND} rm -rf chroot
+ ${LH_ROOT_COMMAND} mv .chroot chroot
+
+ ${LH_ROOT_COMMAND} mount devpts-live -t devpts chroot/dev/pts || true
+ ${LH_ROOT_COMMAND} mount proc-live -t proc chroot/proc
+ ${LH_ROOT_COMMAND} mount sysfs-live -t sysfs chroot/sys
+ else
+ ${LH_ROOT_COMMAND} rm -rf chroot/chroot
+ fi
;;
disabled)
@@ -246,7 +282,25 @@
# Move image
${LH_ROOT_COMMAND} mv chroot/filesystem.squashfs binary/${INITFS}
- ${LH_ROOT_COMMAND} rm -rf chroot/chroot chroot/squashfs.sort
+ if [ -d .chroot ]; then
+ # kill process, which are started during installation and left running
+ kill -15 $(lsof +D chroot -F p|cut -c 2-)
+
+ ${LH_ROOT_COMMAND} umount -f chroot/sys > /dev/null 2>&1 || true
+ ${LH_ROOT_COMMAND} umount -f chroot/proc > /dev/null 2>&1 || true
+ ${LH_ROOT_COMMAND} umount -f chroot/dev/pts > /dev/null 2>&1 || true
+
+ ${LH_ROOT_COMMAND} umount -f chroot.binary/chroot chroot.binary > /dev/null 2>&1 || true
+ ${LH_ROOT_COMMAND} rm -rf chroot.binary chroot.tmp
+ ${LH_ROOT_COMMAND} rm -rf chroot
+ ${LH_ROOT_COMMAND} mv .chroot chroot
+
+ ${LH_ROOT_COMMAND} mount devpts-live -t devpts chroot/dev/pts || true
+ ${LH_ROOT_COMMAND} mount proc-live -t proc chroot/proc
+ ${LH_ROOT_COMMAND} mount sysfs-live -t sysfs chroot/sys
+ else
+ ${LH_ROOT_COMMAND} rm -rf chroot/chroot chroot/squashfs.sort
+ fi
;;
disabled)
@@ -271,11 +325,19 @@
do
if [ "${STAGE}" = "rootfs" ]
then
- rm -rf cache/stages_rootfs
+ if [ -x /usr/bin/rsync ] ;
+ then
+ mkdir -p cache/stages_rootfs
- mkdir -p cache/stages_rootfs
+ ${LH_ROOT_COMMAND} rsync --delete -a binary/"${INITFS}"/filesystem.* cache/stages_rootfs
- ${LH_ROOT_COMMAND} cp -a binary/"${INITFS}"/filesystem.* cache/stages_rootfs
+ else
+ rm -rf cache/stages_rootfs
+
+ mkdir -p cache/stages_rootfs
+
+ ${LH_ROOT_COMMAND} cp -a binary/"${INITFS}"/filesystem.* cache/stages_rootfs
+ fi
if [ -n "${LH_ROOT_COMMAND}" ]
then
diff -rud helpers.1.0~a41+20080325.095215/lh_bootstrap_cache helpers.mod/lh_bootstrap_cache
--- helpers.1.0~a41+20080325.095215/lh_bootstrap_cache 2008-03-25 20:18:22.435041151 +0100
+++ helpers.mod/lh_bootstrap_cache 2008-03-18 08:03:08.851378606 +0100
@@ -56,10 +56,15 @@
Create_lockfile .lock
# Removing old chroot
- rm -rf chroot
+ if [ -x /usr/bin/rsync ];
+ then
+ ${LH_ROOT_COMMAND} rsync --delete -a cache/stages_bootstrap/ chroot
+ else
+ rm -rf chroot
- # Restoring old cache
- ${LH_ROOT_COMMAND} cp -a cache/stages_bootstrap chroot
+ # Restoring old cache
+ ${LH_ROOT_COMMAND} cp -a cache/stages_bootstrap chroot
+ fi
if [ -n "${LH_ROOT_COMMAND}" ]
then
@@ -84,11 +89,16 @@
# Creating lock file
Create_lockfile .lock
- rm -rf cache/stages_bootstrap
+ if [ -x /usr/bin/rsync ];
+ then
+ ${LH_ROOT_COMMAND} rsync --delete -a chroot/ cache/stages_bootstrap
+ else
+ rm -rf cache/stages_bootstrap
- mkdir -p cache
+ mkdir -p cache
- ${LH_ROOT_COMMAND} cp -a chroot cache/stages_bootstrap
+ ${LH_ROOT_COMMAND} cp -a chroot cache/stages_bootstrap
+ fi
if [ -n "${LH_ROOT_COMMAND}" ]
then
diff -rud helpers.1.0~a41+20080325.095215/lh_chroot_cache helpers.mod/lh_chroot_cache
--- helpers.1.0~a41+20080325.095215/lh_chroot_cache 2008-03-25 20:18:22.436049283 +0100
+++ helpers.mod/lh_chroot_cache 2008-03-18 07:55:51.234382767 +0100
@@ -52,11 +52,16 @@
# Creating lock file
Create_lockfile .lock
- # Removing old chroot
- rm -rf chroot
+ if [ -x /usr/bin/rsync ];
+ then
+ ${LH_ROOT_COMMAND} rsync --delete -a cache/stages_chroot/ chroot
+ else
+ # Removing old chroot
+ rm -rf chroot
- # Restoring old cache
- ${LH_ROOT_COMMAND} cp -a cache/stages_chroot chroot
+ # Restoring old cache
+ ${LH_ROOT_COMMAND} cp -a cache/stages_chroot chroot
+ fi
if [ -n "${LH_ROOT_COMMAND}" ]
then
@@ -80,11 +85,15 @@
# Creating lock file
Create_lockfile .lock
- rm -rf cache/stages_chroot
-
- mkdir -p cache
-
- ${LH_ROOT_COMMAND} cp -a chroot cache/stages_chroot
+ if [ -x /usr/bin/rsync ];
+ then
+ mkdir -p cache
+ ${LH_ROOT_COMMAND} rsync --delete -a chroot/ cache/stages_chroot/
+ else
+ rm -rf cache/stages_chroot
+ mkdir -p cache
+ ${LH_ROOT_COMMAND} cp -a chroot cache/stages_chroot
+ fi
if [ -n "${LH_ROOT_COMMAND}" ]
then
diff -rud helpers.1.0~a41+20080325.095215/lh_chroot_hostname helpers.mod/lh_chroot_hostname
--- helpers.1.0~a41+20080325.095215/lh_chroot_hostname 2008-03-25 20:18:22.437046729 +0100
+++ helpers.mod/lh_chroot_hostname 2008-03-25 13:21:48.604050405 +0100
@@ -51,6 +51,7 @@
# Save hostname
mv chroot/bin/hostname chroot/bin/hostname.orig
+ # mv chroot/etc/hostname chroot/etc/hostname.orig
# Create hostname file
echo "localhost.localdomain" > chroot/etc/hostname
@@ -83,6 +84,7 @@
Echo_message "Deconfiguring file /bin/hostname"
# Restore hostname file
+ #mv chroot/etc/hostname.orig chroot/etc/hostname
mv chroot/bin/hostname.orig chroot/bin/hostname
# Removing stage file
diff -rud helpers.1.0~a41+20080325.095215/lh_chroot_sources helpers.mod/lh_chroot_sources
--- helpers.1.0~a41+20080325.095215/lh_chroot_sources 2008-03-25 20:18:22.444045579 +0100
+++ helpers.mod/lh_chroot_sources 2008-03-17 18:48:16.211380513 +0100
@@ -150,11 +150,11 @@
# Installing aptitude
if [ "${LH_APT}" = "aptitude" ] && [ ! -x /usr/bin/aptitude ]
then
- Chroot "apt-get ${APT_OPTIONS} update"
+ Chroot "apt-get ${APT_OPTIONS} update > /dev/null"
Chroot "apt-get ${APT_OPTIONS} install aptitude"
fi
- Apt update
+ Apt update > /dev/null
Apt upgrade
Apt dist-upgrade
@@ -162,7 +162,7 @@
if [ -n "${LH_KEYRING_PACKAGES}" ]
then
Apt --force-yes install "${LH_KEYRING_PACKAGES}"
- Apt update
+ Apt update > /dev/null
fi
if [ "${LH_CACHE_INDICES}" = "enabled" ]
@@ -176,7 +176,7 @@
cp -f chroot/var/cache/apt/srcpkgcache.bin cache/indices_bootstrap
cp -f chroot/var/lib/apt/lists/*_Packages cache/indices_bootstrap
- cp -f chroot/var/lib/apt/lists/*_Sources cache/indices_bootstrap
+ #cp -f chroot/var/lib/apt/lists/*_Sources cache/indices_bootstrap
cp -f chroot/var/lib/apt/*_Release* cache/indices_bootstrap
fi
fi
@@ -255,7 +255,7 @@
fi
# Updating indices
- Apt update
+ Apt update > /dev/null
fi
# Cleaning apt packages cache
lh_functions.sh
Description: application/shellscript
_______________________________________________ debian-live-devel mailing list [email protected] http://lists.alioth.debian.org/mailman/listinfo/debian-live-devel
