Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2021-03-08 Thread ValdikSS 
Debootstrap is used in Debian ISO installer (debian-installer, d-i) for 
initial target rootfs bootstrapping. With sync enabled, this takes 
hundreds of minutes to unpack and configure base system on a HDD.


With eatmydata it takes about a minute.

What could be done to introduce eatmydata into debootstrap? Are propsed 
patches above (in message 50 and before) are not correct or not conform 
to Debian style or policy? If so, what should be improved?






OpenPGP_signature
Description: OpenPGP digital signature

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2019-09-16 Thread Thorsten Glaser 
Version: 1.0.115

Hi everyone,

I’m appalled that this feature request is still open…

I’ve got a need to do this without patching debootstrap, and have had
success (although I did not time it, but someone else can ;-) with this
sequence of commands (assuming dpkg-deb is available ― which my script
checks earlier):

safe_PATH=/bin:/sbin:/usr/bin:/usr/sbin
# […]
case $TERM in
(Eterm|Eterm-color|ansi|cons25|cons25-debian|cygwin|dumb|hurd|linux|mach|mach-bold|mach-color|mach-gnu|mach-gnu-color|pcansi|rxvt|rxvt-basic|rxvt-m|rxvt-unicode|rxvt-unicode-256color|screen|screen-256color|screen-256color-bce|screen-bce|screen-s|screen-w|screen.xterm-256color|sun|vt100|vt102|vt220|vt52|wsvt25|wsvt25m|xterm|xterm-256color|xterm-color|xterm-debian|xterm-mono|xterm-r5|xterm-r6|xterm-vt220|xterm-xfree86)
# list from ncurses-base (6.1+20181013-2+deb10u1)
;;
(screen.*|screen-*)
# aliases possibly from ncurses-term
TERM=screen ;;
(rxvt.*|rxvt-*)
# let’s hope…
TERM=rxvt ;;
(xterm.*|xterm-*)
# …this works…
TERM=xterm ;;
(linux.*)
# …probably
TERM=linux ;;
(*)
die "Your terminal type '$TERM' is not supported by ncurses-base." \
'Maybe run this script in GNU screen?' ;;
esac
# […]
eatmydata debootstrap --arch=arm64 --include=eatmydata --no-merged-usr \
--force-check-gpg --verbose --foreign buster "$mpt" \
http://deb.debian.org/debian sid
# script specified here as it’s normally what buster symlinks to,
# to achieve compatibility with more host distros
# we need this early
(
set -e
cd "$mpt"
for archive in var/cache/apt/archives/*eatmydata*.deb; do
dpkg-deb --fsys-tarfile "$archive" >a
tar -xkf a
done
rm -f a
) || die 'failure extracting eatmydata early'
# the user can delete this later, from the booted system (or apt will)
cp /usr/bin/qemu-aarch64-static "$mpt/usr/bin/" || die 'cp failed'
echo >&2 'I: second stage bootstrap (under emulation), slooow…'
mount -t tmpfs swap "$mpt/dev/shm" || die 'mount /dev/shm failed'
mount -t proc  proc "$mpt/proc" || die 'mount /proc failed'
mount -t tmpfs swap "$mpt/tmp" || die 'mount /tmp failed'
chroot "$mpt" /usr/bin/env -i LC_ALL=C.UTF-8 HOME=/root PATH="$safe_PATH" \
TERM="$TERM" /usr/bin/eatmydata /debootstrap/debootstrap --second-stage || \
die 'debootstrap (second stage) failed'


Of course, “success” here means it does not error out… ☻ (also, as it’s
run under emulation it’s very slooow anyway…)

bye,
//mirabilos
-- 
tarent solutions GmbH
Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/
Tel: +49 228 54881-393 • Fax: +49 228 54881-235
HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

**

Mit der tarent Academy bieten wir auch Trainings und Schulungen in den
Bereichen Softwareentwicklung, Agiles Arbeiten und Zukunftstechnologien an.

Besuchen Sie uns auf www.tarent.de/academy. Wir freuen uns auf Ihren Kontakt.

**

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2018-04-08 Thread Hideki Yamane 
Hi,

On Sat, 16 Feb 2013 16:54:52 -0700 Bob Proulx  wrote:
> I have also been suffering with the much slower dpkg on Sid.  Today I
> performed a test of a default debootstrap chroot and had this
> benchmark data.  I am using a local full mirror so that network
> performance is not significant in the measurement.  This is using the
> stock Sid version of everything up to date as of today operating on an
> ext3 filesystem.
> 
>   # time debootstrap wheezy testinstall 
> http://localmirror/ftp.us.debian.org/debian
>   ...
>   real2m58.577s
>   user0m49.639s
>   sys 0m8.749s
> 
> And then using the patched debootstrap script Phillip Susi suggested I
> get this result:
> 
>   # time debootstrap wheezy testinstall 
> http://localmirror/ftp.us.debian.org/debian
>   ...
>   real1m17.680s
>   user0m49.879s
>   sys 0m7.988s
> 
> Wall clock time is more than 2x faster for me when avoiding the
> unnecessary fsync() calls.  Ideally I would like to see dpkg handle
> this issue.  But since that doesn't seem to be happening it would be
> nice if the rest of the ecosystem could work around it.
> 
> Here is the same test again on an ext4 filesystem on the same system
> and the stock scripts.
> 
>   # time debootstrap wheezy testinstall 
> http://localmirror/ftp.us.debian.org/debian
>   ...
>   real2m55.402s
>   user0m52.107s
>   sys 0m7.888s
> 
> And again with the eatmydata patched script:
> 
>   # time debootstrap wheezy testinstall 
> http://localmirror/ftp.us.debian.org/debian
>   ...
>   real1m4.819s
>   user0m48.455s
>   sys 0m6.752s
> 
> That is 2.7x faster on ext4.

 I've tested it but could not get good result as yours.

 - Use SSD and USB2.0 connected HDD
 - both ext4 formatted
 - debootstrap is patched with https://bugs.debian.org/844118


Without eatmydata on SSD
---
real2m10.057s
user1m47.776s
sys 0m8.060s

With eatmydata on SSD
---
real2m8.640s
user1m47.546s   
sys 0m8.049s  

Without eatmydata on HDD
---
real2m25.136s
user1m47.758s
sys 0m8.603s

With eatmydata on HDD
---
real2m26.790s
user1m48.699s
sys 0m8.470s


 So I doubt using eatmydata.


-- 
Regards,

 Hideki Yamane henrich @ debian.org/iijmio-mail.jp

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2013-12-26 Thread Marc Haber 

On Mon, Apr 01, 2013 at 10:14:16AM -0400, Phillip Susi wrote:
 On 3/31/2013 4:17 PM, Marc Haber wrote:
  The following patch introduces an --include-early option which
  allows to introduce eatmydata early enough:
 
 Is this intended to be applied instead of, or on top of my initial
 patch?  If the former ( which I'm guessing it is ), I don't see where
 in-target is modified to *use* it?

This is intended to be used instead of your patch, and the intended
use is

eatmydata debootstrap --include-early=...

which will invoke deboostrap under eatymdata which will cause all
binaries to be started under eatmydata and additionally will cause
eatmydata to be installed into the chroot at the earliest convenience
so that calls started inside the chroot actually find libeatmydata.

Greetings
Marc

-- 
-
Marc Haber | I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things.Winona Ryder | Fon: *49 621 31958061
Nordisch by Nature |  How to make an American Quilt | Fax: *49 621 31958062


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2013-04-01 Thread Phillip Susi 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 3/31/2013 4:17 PM, Marc Haber wrote:
 The following patch introduces an --include-early option which
 allows to introduce eatmydata early enough:

Is this intended to be applied instead of, or on top of my initial
patch?  If the former ( which I'm guessing it is ), I don't see where
in-target is modified to *use* it?


-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRWZY4AAoJEJrBOlT6nu75s1oH/isdxtsbIsI4jcp/sVNSoIXt
XKS+2e/4qoQH3pnAeUvRWqYidgoCfZZXMK08FBcD2vWt+dgCBvroXTK+h3MU7bav
+1QiyEr9P0ba/w5qbCNf84YjOLd3AXSTBmi1vlQDC4tXIzGi2KyBM5gWKDLKL7i1
sPgJFdiDb8AYOI0WB5fCRr/ba2fQgGW81TdIK7aTE2/2IOrlisQ6NyNhahglNJ2G
qHODM5vjo0SkBvHqlhM+npOsKdva1l6IMRHlfoonGtx+o01YzNlC4SnonIzRvx8S
5+vhXIzRYCH/WeY7b45ygdY/i1Q8uJFxecQn2Kcw2GkbEZeaPHlql9as1kO4HP0=
=vyZQ
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2013-03-31 Thread Marc Haber 
 Debootstrap is very, very slow. I watched a server install crawl by
 for something like an hour at a rate of about 1 package per second. A
 simple debootstrap chroot takes 10 minutes to setup, with a
 predownloaded tarball. This slowness is due to dpkg making excessive
 use of fsync(). Requests to fix dpkg in the past have been rejected
 and I was told to use eatmydata. After patching debootstrap to add
 eatmydata to the required list, and activate it during the second
 stage install, the time to construct the chroot dropped from 10m to
 2m. This should also make installing new systems MUCH faster.

The following patch introduces an --include-early option which allows
to introduce eatmydata early enough:

--- /usr/sbin/debootstrap   2013-03-22 07:04:12.0 +0100
+++ tmp/debootstrap/debootstrap 2013-03-31 14:17:26.425276613 +0200
@@ -40,6 +40,7 @@
 EXTRACTOR_OVERRIDE=
 UNPACK_TARBALL=
 ADDITIONAL=
+ADDITIONAL_EARLY=
 EXCLUDE=
 VERBOSE=
 CERTIFICATE=
@@ -94,6 +95,8 @@
[ --arch=powerpc ]

   --include=A,B,Cadds specified names to the list of base packages
+  --include-early=A,B,C  adds specified names to the list of required 
packages
+ (will be unpacked before chrooting in for the 
first time)
   --exclude=A,B,Cremoves specified packages from the list
   --components=A,B,C use packages from the listed components of the
  archive
@@ -259,6 +262,18 @@
fi
ADDITIONAL=$(echo $ADDITIONAL | tr ,  )
;;
+   --include-early|--include-early=?*)
+   if [ $1 = --include-early -a -n $2 ]; then
+   ADDITIONAL_EARLY=$2
+   shift 2
+elif [ $1 != ${1#--include-early=} ]; then
+   ADDITIONAL_EARLY=${1#--include-early=}
+   shift 1
+   else
+   error 1 NEEDARG option requires an argument %s $1
+   fi
+   ADDITIONAL_EARLY=$(echo $ADDITIONAL_EARLY | tr ,  )
+   ;;
--exclude|--exclude=?*)
if [ $1 = --exclude -a -n $2 ]; then
EXCLUDE=$2
@@ -561,6 +576,7 @@
work_out_debs

base=$(without $base $ADDITIONAL $EXCLUDE)
+   required=$required $ADDITIONAL_EARLY

if [ $RESOLVE_DEPS = true ]; then
requiredX=$(echo $(echo $required | tr ' ' '\n' | sort | uniq)) 

This also does not require all distribution scripts to be adapted.

Here is the man page patch:

--- debootstrap.8.orig  2013-03-31 21:40:29.694855662 +0200
+++ debootstrap.8   2013-03-31 21:41:19.114966193 +0200
@@ -54,6 +54,12 @@
 Comma separated list of packages which will be added to download and extract
 lists.
 .IP
+.IP \fB\-\-include-early=alpha,beta\fP
+Comma separated list of packages which will be added to download and extract
+lists. These packages are already unpacked during the first stage
+which allows, for example, eatmydata to be used early enough in the
+process.
+.IP
 .IP \fB\-\-exclude=alpha,beta\fP
 Comma separated list of packages which will be removed from download and
 extract lists.


With this patch, one can use

eatmydata debootstrap --include-early=eatmydata ...

to get eatmydata. With strace -esync,fsync, I verified that this
actually only issues about 20 fsync calls in the final stage of
debootstrap.

On a rotating hard disk, this reduces the time needed to set up a sid
bootstrap from 10 minutes to 4 minutes. On an SSD, the time difference
is negligible though.

Greetings
Marc

-- 
-
Marc Haber | I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany  |  lose things.Winona Ryder | Fon: *49 621 31958061
Nordisch by Nature |  How to make an American Quilt | Fax: *49 621 31958062


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2013-02-16 Thread Bob Proulx 
Phillip Susi wrote:
 After patching debootstrap to add eatmydata to the required list,
 and activate it during the second stage install, the time to
 construct the chroot dropped from 10m to 2m. This should also make
 installing new systems MUCH faster.

I have also been suffering with the much slower dpkg on Sid.  Today I
performed a test of a default debootstrap chroot and had this
benchmark data.  I am using a local full mirror so that network
performance is not significant in the measurement.  This is using the
stock Sid version of everything up to date as of today operating on an
ext3 filesystem.

  # time debootstrap wheezy testinstall 
http://localmirror/ftp.us.debian.org/debian
  ...
  real2m58.577s
  user0m49.639s
  sys 0m8.749s

And then using the patched debootstrap script Phillip Susi suggested I
get this result:

  # time debootstrap wheezy testinstall 
http://localmirror/ftp.us.debian.org/debian
  ...
  real1m17.680s
  user0m49.879s
  sys 0m7.988s

Wall clock time is more than 2x faster for me when avoiding the
unnecessary fsync() calls.  Ideally I would like to see dpkg handle
this issue.  But since that doesn't seem to be happening it would be
nice if the rest of the ecosystem could work around it.

Here is the same test again on an ext4 filesystem on the same system
and the stock scripts.

  # time debootstrap wheezy testinstall 
http://localmirror/ftp.us.debian.org/debian
  ...
  real2m55.402s
  user0m52.107s
  sys 0m7.888s

And again with the eatmydata patched script:

  # time debootstrap wheezy testinstall 
http://localmirror/ftp.us.debian.org/debian
  ...
  real1m4.819s
  user0m48.455s
  sys 0m6.752s

That is 2.7x faster on ext4.

Bob


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2013-02-16 Thread Phillip Susi 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/16/2013 06:54 PM, Bob Proulx wrote:
 # time debootstrap wheezy testinstall
 http://localmirror/ftp.us.debian.org/debian ... real2m58.577s 
 user0m49.639s sys 0m8.749s

Wow, that's pretty fast.  I was testing on a server with a raid5 array
and was seeing nearly 10m original time, down to 1-2m patched.  And
that's with a pre built debootstrap tarball so nothing needed downloaded.



-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRICQUAAoJEJrBOlT6nu75g+UIANj9vklDF3cYmxrsxTs5ImIB
ev0IsbrjLz7BePRG/OCusG/V1i2Q4R8XHEfHxJ3vkP/aAh/q7uqsDXJ6ozjSG7P/
32BmRjJx9VBzmKVQIEFfjbFbqIEr+3SBRMAWF5eCvnGXd0rXhBjcslSJ4xbVVq/M
IuwXw8clwgo2POSWKxNmK5/Rk8EJZddRil6ioGeOKq3hRr5jDAx7W/x30dnYDtQE
ImjTCP+n0rDd97VncblRA3Opsurzeqjqhs5YRHS7CFumph5yUH5GuVNQrm71Ztc7
T34g8UnkDUnPQ/YWrM1UExpokwkauEcxxtwn8RkS35Ie+AKBINotykDBspoXbHA=
=/i4U
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2013-02-16 Thread Bob Proulx 
Phillip Susi wrote:
 Bob Proulx wrote:
  # time debootstrap wheezy testinstall
  http://localmirror/ftp.us.debian.org/debian ... real2m58.577s
  user0m49.639s sys 0m8.749s

 Wow, that's pretty fast.

2m58s was the slow time.  1m4.819s was the fast time.  :-)

 I was testing on a server with a raid5 array and was seeing nearly
 10m original time, down to 1-2m patched.

This is on a relatively fast machine.  But the benefits will be more
dramatic on machines with slower I/O bandwidth for disk storage.

 And that's with a pre built debootstrap tarball so nothing needed
 downloaded.

With --download-only I assume?  Or otherwise is fine.  Sounds good.

Bob


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2013-02-15 Thread Phillip Susi 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Package: debootstrap
Version: 1.0.46

Debootstrap is very, very slow. I watched a server install crawl by
for something like an hour at a rate of about 1 package per second. A
simple debootstrap chroot takes 10 minutes to setup, with a
predownloaded tarball. This slowness is due to dpkg making excessive
use of fsync(). Requests to fix dpkg in the past have been rejected
and I was told to use eatmydata. After patching debootstrap to add
eatmydata to the required list, and activate it during the second
stage install, the time to construct the chroot dropped from 10m to
2m. This should also make installing new systems MUCH faster.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRHlSFAAoJEJrBOlT6nu75+dcIAK/GwMGqqZDQJRLS7//MoRYH
DXFJa4PExLKnvyDvqONMV/gRNY7fc6E8kuEERZeySflgYzdxU28CwNCgB2+UFydh
DiyARnEv+fXmjnSS0YtvOt8ZTr2z0UuMXpuMGz0+Yca88gGEnlPiB0tqcbQx0mrG
GpGBnYLKpdk/mz5tsRo+LFgAKC8fsp6ievf+6w7vIsjrkWkuL5i+3lPBjmGlBtZ6
Bfaxk31Px9OBCpI1sX/NFNilQl++MAK2Ejm6FMblRSW7aWJXROHIGJzz74wQKqjP
Kweqf6bqrcKQnBlKwKDooniIEqbYs0ewgF9jqkx+dz60taBPs+2swKtv2d82pRQ=
=MoT3
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2013-02-15 Thread Cyril Brulebois 
Phillip Susi ps...@ubuntu.com (15/02/2013):
 This should also make installing new systems MUCH faster.

FWIW:
| base-installer (1.121) unstable; urgency=low
| 
|   [ Colin Watson ]
|   * Merge from Ubuntu:
| - Run dpkg with --force-unsafe-io during installation; syncing is
|   unnecessary in this context and can slow things down quite a bit
|   (closes: #605384).
| […]
| 
|  -- Joey Hess jo...@debian.org  Wed, 24 Aug 2011 19:11:09 -0400

Mraw,
KiBi.


signature.asc
Description: Digital signature

Bug#700633: Debootstrap is very slow. Please use eatmydata to fix this.

2013-02-15 Thread Phillip Susi 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/15/2013 09:09 PM, Cyril Brulebois wrote:


- --force-unsafe-io only eliminates *some* of the syncs, and seems to
only be applied in the latter part of the installation that takes
place after debootstrap.

I started working on a patch to dpkg to add another --force option to
eliminate the rest of the syncs a year or two ago and was told on the
dpkg mailing list to just use libeatmydata, as it would also make sure
any syncs done by the [pre|post][inst|rm] scripts are also disabled.


-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQEcBAEBAgAGBQJRHv+pAAoJEJrBOlT6nu75t28H/0tfko7zwRQ0Y1hWN8hE1TXj
ZaDxoIZ1aQ6fVAm4uow+7vXiP4TBPnR61sHvs7mO791sNlVutG7Lz6D4XG7dWmID
N/hKRbX/wYz02rEeuEw2z3MrynfWq53fX9I+PBn0aFyBNtLM0oHMGoaSDw9TRnbB
/+9+v+nLd//dA8gvXGLsx97ildV6uS4ZtI814l3uC3DnhrSBvqP4TVOtaGmiGe7m
NZdb5FyfSt3vGBeMrFsuAHDl8IyQyWX6gDcZvUuz7qcSXHS2NCrAAs0IsoM41bpN
3oZtVJn6uYB/rvkR4DfyYh8FerhiQ9x/Qx96Dmxz6cE3+CrGoYdY3FLGzsB0ydQ=
=mn+R
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org