Bug#864777: initramfs-tools: Please support a tmpfs boot mode

[Benjamin Drung]

Hi Peter,

Am Donnerstag, den 29.06.2017, 15:55 -0400 schrieb Peter Colberg:
> Hi Benjamin,
> 
> On Wed, Jun 14, 2017 at 05:47:54PM +0200, Benjamin Drung wrote:
> > Package: initramfs-tools
> > Version: 0.130
> > Severity: normal
> > Tags: patch
> > 
> > Hi,
> > 
> > The tmpfs boot mode allows one to operate a disk-less live system
> > by
> > downloading a root tarball (that can be created with debootstrap)
> > and
> > extracting it to a tmpfs root partition.
> > 
> > The tmpfs boot mode is similar to the nfs boot mode, but it does
> > not
> > rely on any external service (after booting). We use this boot mode
> > for
> > our compute nodes. I have attached a patch to support this boot
> > mode and
> > tested it with qemu and on real hardware.
> 
> Please take a look at the package live-boot (and the man page of the
> same name in live-boot-doc). Specifically, the parameter fetch=URL
> should satisfy your requirements and consume less resources.
> 
> live-boot creates an overlay filesystem backed by a squashfs image,
> which means it likely uses much less RAM than copying root to tmpfs.
> (For my system, squashfs reduces the required memory to a third.)

I took a look at the live-boot package and we will evaluate the RAM
savings and performance penalty. Expect a patch for live-boot to
support live-{top,premount,bottom} hooks.

Some statistics: Our xz compressed root tarball is around 180 MB in
size and consumes 690 MB extracted. Our initrd uses 9 MB. The initrd
from live-boot uses 17 MB. Let's see if the live-boot initrd can be
stripped down as well.

For now, I updated my patch to add an option to enable/disable support
for the tmpfs boot mode. Updated patch is attached.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.dr...@profitbricks.com
URL: https://www.profitbricks.de

Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss, Matthias SteinbergFrom 500c0566ef08a460b666c881b6d5a120b01501f4 Mon Sep 17 00:00:00 2001
From: Benjamin Drung 
Date: Mon, 12 Jun 2017 15:31:58 +0200
Subject: [PATCH 1/2] Fix indentation (spaces to tabs)

---
 init | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/init b/init
index a88a64c..035469b 100755
--- a/init
+++ b/init
@@ -85,7 +85,7 @@ for x in $(cat /proc/cmdline); do
 		if [ -z "${BOOT}" ] && [ "$ROOT" = "/dev/nfs" ]; then
 			BOOT=nfs
 		fi
-;;
+		;;
 	rootflags=*)
 		ROOTFLAGS="-o ${x#rootflags=}"
 		;;
@@ -115,7 +115,7 @@ for x in $(cat /proc/cmdline); do
 	resume=*)
 		RESUME="${x#resume=}"
 		case $RESUME in
-	UUID=*)
+		UUID=*)
 			RESUME="/dev/disk/by-uuid/${RESUME#UUID=}"
 		esac
 		;;
-- 
2.14.1

From c236415782ea0171a64049d273b1fab557ba6698 Mon Sep 17 00:00:00 2001
From: Benjamin Drung 
Date: Wed, 7 Jun 2017 18:04:47 +0200
Subject: [PATCH 2/2] Add tmpfs boot mode

The tmpfs boot mode allows one to operate a disk-less live system by downloading
a root tarball (that can be created with debootstrap) and extracting it to a
tmpfs root partition.

Closes: #864777
Signed-off-by: Benjamin Drung 
---
 conf/initramfs.conf |  22 
 hooks/tmpfs |  39 +
 init|  14 -
 initramfs-tools.8   |  23 
 initramfs.conf.5|  14 +
 mkinitramfs |   1 +
 scripts/tmpfs   | 158 
 7 files changed, 259 insertions(+), 12 deletions(-)
 create mode 100755 hooks/tmpfs
 create mode 100644 scripts/tmpfs

diff --git a/conf/initramfs.conf b/conf/initramfs.conf
index f0f5071..4af51e4 100644
--- a/conf/initramfs.conf
+++ b/conf/initramfs.conf
@@ -62,3 +62,25 @@ DEVICE=
 
 NFSROOT=auto
 
+#
+# tmpfs section of the config.
+#
+
+#
+# TMPFS: [ y | n ]
+#
+# Enable tmpfs boot mode support (i.e. copy required files into initramfs).
+#
+
+TMPFS=n
+
+#
+# ROOTSIZE: ...
+#
+# The size of the tmpfs root mount in bytes, and rounded up to entire pages.
+# This option accepts a suffix % to limit the instance to that percentage of
+# your physical RAM or a suffix k, m or g for Ki, Mi, Gi (binary kilo (kibi),
+# binary mega (mebi) and binary giga (gibi)).
+#
+
+ROOTSIZE=50%
diff --git a/hooks/tmpfs b/hooks/tmpfs
new file mode 100755
index 000..49bbca0
--- /dev/null
+++ b/hooks/tmpfs
@@ -0,0 +1,39 @@
+#!/bin/sh
+
+PREREQ=""
+
+prereqs()
+{
+	echo "$PREREQ"
+}
+
+case $1 in
+# get pre-requisites
+prereqs)
+	prereqs
+	exit 0
+	;;
+esac
+
+# Hook for supporting tmpfs boot mode if requested by TMPFS="y"
+if [ "$TMPFS" != "y" ] && [ "$TMPFS" != "Y" ]; then
+	exit 0
+fi
+
+. /usr/share/initramfs-tools/hook-functions
+
+# GNU tar is needed for xattrs support
+if test -e /bin/tar; then
+	test -z "$DESTDIR" || rm -f ${DESTDIR}/bin/tar
+	copy_exec /bin/tar
+fi
+
+# pixz is used to speed up the decompression
+if test -e 

Bug#864777: initramfs-tools: Please support a tmpfs boot mode

[Peter Colberg]

Hi Benjamin,

On Wed, Jun 14, 2017 at 05:47:54PM +0200, Benjamin Drung wrote:
> Package: initramfs-tools
> Version: 0.130
> Severity: normal
> Tags: patch
> 
> Hi,
> 
> The tmpfs boot mode allows one to operate a disk-less live system by
> downloading a root tarball (that can be created with debootstrap) and
> extracting it to a tmpfs root partition.
> 
> The tmpfs boot mode is similar to the nfs boot mode, but it does not
> rely on any external service (after booting). We use this boot mode for
> our compute nodes. I have attached a patch to support this boot mode and
> tested it with qemu and on real hardware.

Please take a look at the package live-boot (and the man page of the
same name in live-boot-doc). Specifically, the parameter fetch=URL
should satisfy your requirements and consume less resources.

live-boot creates an overlay filesystem backed by a squashfs image,
which means it likely uses much less RAM than copying root to tmpfs.
(For my system, squashfs reduces the required memory to a third.)

Regards,
Peter

Bug#864777: initramfs-tools: Please support a tmpfs boot mode

[Benjamin Drung]

Package: initramfs-tools
Version: 0.130
Severity: normal
Tags: patch

Hi,

The tmpfs boot mode allows one to operate a disk-less live system by
downloading a root tarball (that can be created with debootstrap) and
extracting it to a tmpfs root partition.

The tmpfs boot mode is similar to the nfs boot mode, but it does not
rely on any external service (after booting). We use this boot mode for
our compute nodes. I have attached a patch to support this boot mode and
tested it with qemu and on real hardware.

I am willing to maintain this boot mode and fix bugs in this part.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.dr...@profitbricks.com
Web: https://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Achim Weiss.
>From 7cb2a6e5fb601edbad7359157f135fe17194a49d Mon Sep 17 00:00:00 2001
From: Benjamin Drung 
Date: Mon, 12 Jun 2017 15:31:58 +0200
Subject: [PATCH 1/2] Fix indentation (spaces to tabs)

---
 init | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/init b/init
index 1334dca..4100d93 100755
--- a/init
+++ b/init
@@ -81,7 +81,7 @@ for x in $(cat /proc/cmdline); do
if [ -z "${BOOT}" ] && [ "$ROOT" = "/dev/nfs" ]; then
BOOT=nfs
fi
-;;
+   ;;
rootflags=*)
ROOTFLAGS="-o ${x#rootflags=}"
;;
@@ -111,7 +111,7 @@ for x in $(cat /proc/cmdline); do
resume=*)
RESUME="${x#resume=}"
case $RESUME in
-   UUID=*)
+   UUID=*)
RESUME="/dev/disk/by-uuid/${RESUME#UUID=}"
esac
;;
-- 
2.11.0

>From d8c7dd1922c9ba8bb63156843c4daf869b765c6b Mon Sep 17 00:00:00 2001
From: Benjamin Drung 
Date: Wed, 7 Jun 2017 18:04:47 +0200
Subject: [PATCH 2/2] Add tmpfs boot mode

The tmpfs boot mode allows one to operate a disk-less live system by downloading
a root tarball (that can be created with debootstrap) and extracting it to a
tmpfs root partition.
---
 conf/initramfs.conf |  14 +
 hooks/tmpfs |  36 
 init|  14 -
 initramfs-tools.8   |  21 ---
 initramfs.conf.5|   9 +++
 scripts/tmpfs   | 158 
 6 files changed, 241 insertions(+), 11 deletions(-)
 create mode 100755 hooks/tmpfs
 create mode 100644 scripts/tmpfs

diff --git a/conf/initramfs.conf b/conf/initramfs.conf
index f0f5071..d75fe04 100644
--- a/conf/initramfs.conf
+++ b/conf/initramfs.conf
@@ -62,3 +62,17 @@ DEVICE=
 
 NFSROOT=auto
 
+#
+# tmpfs section of the config.
+#
+
+#
+# ROOTSIZE: ...
+#
+# The size of the tmpfs root mount in bytes, and rounded up to entire pages.
+# This option accepts a suffix % to limit the instance to that percentage of
+# your physical RAM or a suffix k, m or g for Ki, Mi, Gi (binary kilo (kibi),
+# binary mega (mebi) and binary giga (gibi)).
+#
+
+ROOTSIZE=50%
diff --git a/hooks/tmpfs b/hooks/tmpfs
new file mode 100755
index 000..687000f
--- /dev/null
+++ b/hooks/tmpfs
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+PREREQ=""
+
+prereqs()
+{
+   echo "$PREREQ"
+}
+
+case $1 in
+# get pre-requisites
+prereqs)
+   prereqs
+   exit 0
+   ;;
+esac
+
+# Hook for supporting tmpfs boot mode
+
+. /usr/share/initramfs-tools/hook-functions
+
+# GNU tar is needed for xattrs support
+if test -e /bin/tar; then
+   test -z "$DESTDIR" || rm -f ${DESTDIR}/bin/tar
+   copy_exec /bin/tar
+fi
+
+# pixz is used to speed up the decompression
+if test -e /usr/bin/pixz; then
+   copy_exec /usr/bin/pixz
+fi
+
+# libnss_dns is needed for resolving hosts by name
+if test -e /lib/x86_64-linux-gnu/libnss_dns.so.2; then
+   copy_exec /lib/x86_64-linux-gnu/libnss_dns.so.2
+fi
diff --git a/init b/init
index 4100d93..735afad 100755
--- a/init
+++ b/init
@@ -78,8 +78,15 @@ for x in $(cat /proc/cmdline); do
;;
root=*)
ROOT=${x#root=}
-   if [ -z "${BOOT}" ] && [ "$ROOT" = "/dev/nfs" ]; then
-   BOOT=nfs
+   if [ -z "${BOOT}" ]; then
+   case ${ROOT} in
+   /dev/nfs)
+   BOOT=nfs
+   ;;
+   http*)
+   BOOT=tmpfs
+   ;;
+   esac
fi
;;
rootflags=*)
@@ -96,6 +103,9 @@ for x in $(cat /proc/cmdline); do
;;
esac
;;
+   rootsize=*)
+   ROOTSIZE="${x#rootsize=}"
+   ;;
nfsroot=*)
NFSROOT="${x#nfsroot=}"
;;
diff --git a/initramfs-tools.8