Bug#871835: speed up for debootstrap

[Roger Shimizu]

Dear Thomas,

On Wed, Jan 2, 2019 at 7:51 PM Thomas Lange
 wrote:
>
> > On Sun, 23 Dec 2018 22:52:36 +0900, Roger Shimizu 
> >  said:
>
> > On Sat, Aug 12, 2017 at 1:54 PM Thomas Lange
> >  wrote:
> >>
> >> old version 1.0.89
> >> real1m26.940s
> >> user1m24.180s
> >> sys 0m2.420s
> >>
> >>
> >> including the patches
> >> real0m39.481s
> >> user0m50.444s
> >> sys 0m2.748s
>
> > have you compared with cdebootstrap?
> > - https://tracker.debian.org/pkg/cdebootstrap
> cdebootstrap takes about 30 sec real time and 40 sec user time.

Really thanks for your test result, and the patches!

Since cdebootstrap usually install smaller set of packages, I think
they're now pretty close at speed.

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Bug#871835: speed up for debootstrap

[Roger Shimizu]

On Sat, Aug 12, 2017 at 1:54 PM Thomas Lange
 wrote:
>
> old version 1.0.89
> real1m26.940s
> user1m24.180s
> sys 0m2.420s
>
>
> including the patches
> real0m39.481s
> user0m50.444s
> sys 0m2.748s

have you compared with cdebootstrap?
- https://tracker.debian.org/pkg/cdebootstrap

Just curious..

Cheers,
-- 
Roger Shimizu, GMT +9 Tokyo
PGP/GPG: 4096R/6C6ACD6417B3ACB1

Bug#871835: speed up for debootstrap

[Thomas Lange]

I repeated my tests with debootstrap 1.0.106 on buster and everything
went OK. The directory that is generated using the original functions
file and the patched one are the same. Here are the debootstrap calls
I've tested so far:

debootstrap stretch
debootstrap --variant=minbase stretch
debootstrap --variant=buildd stretch
debootstrap --foreign stretch
debootstrap --arch i386 --variant=buildd stretch
debootstrap --arch i386 stretch
debootstrap --include=top,git,tig stretch
debootstrap --exclude=dmidecode,tasksel,ncurses-bin stretch

debootstrap buster
debootstrap --variant=minbase buster
debootstrap --variant=buildd buster
debootstrap --foreign buster
debootstrap --arch i386 --variant=buildd buster
debootstrap --arch i386 buster
debootstrap --include=top,git,tig buster
debootstrap --exclude=dmidecode,tasksel,ncurses-bin buster


If you have more things to test, just let me know.

-- 
regards Thomas

Bug#871835: speed up for debootstrap

[Hideki Yamane]

On Fri, 27 Jul 2018 11:25:40 +0200
Cyril Brulebois  wrote:
> For some reason it seems I didn't follow up to this bug report but I'm
> pretty sure I did reply on debian-boot@ in some way: we need automated
> tests before and after those patches. And we need to make sure they
> yield the exact same results. Please don't merge the whole stack of
> patches without such checks.

 I heard about Thomas has set his test system and works well.
 It'd be better to share its info, then.


-- 
Hideki Yamane 

Bug#871835: speed up for debootstrap

[Cyril Brulebois]

Hi,

Hideki Yamane  (2018-07-27):
>  As prepared MR in 
> https://salsa.debian.org/installer-team/debootstrap/merge_requests/19
>  I'll put it into 1.0.108, _after DebConf_ since I don't have GPG key on
>  my laptop (I have the new one but it's not in keyring now).
> 
>  If some one want to push 1.0.107 into repository, it's nice.

For some reason it seems I didn't follow up to this bug report but I'm
pretty sure I did reply on debian-boot@ in some way: we need automated
tests before and after those patches. And we need to make sure they
yield the exact same results. Please don't merge the whole stack of
patches without such checks.



Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#871835: speed up for debootstrap

[Hideki Yamane]

Hi,

 As prepared MR in 
https://salsa.debian.org/installer-team/debootstrap/merge_requests/19
 I'll put it into 1.0.108, _after DebConf_ since I don't have GPG key on
 my laptop (I have the new one but it's not in keyring now).

 If some one want to push 1.0.107 into repository, it's nice.


-- 
Hideki Yamane 

Bug#871835: speed up for debootstrap

[Thomas Lange]

I forget something. During the run of debootstrap, I observed via
repeated ps -efl calls that often perl is running for a longer
time. Then I tried to use strace to see which exec(ve) calls takes a
long time. But I didn't managed to find out, instead I saw read calls
taking long. So I decided to look at the perl code and what it was
reading.
Using strace I could also see which perl calls where executed several
times.
 
-- 
regards Thomas

Bug#871835: speed up for debootstrap

[Thomas Lange]

> On Thu, 7 Sep 2017 18:28:33 +0200, Tiago Ilieve  
> said:

> Do you mind to share how you identified those slower parts of the
> code? Have you used a profiler or a similar tool?
I didn't used a profiler. I just look at the code and saw that the
/var/lib/dpkg/available files is read several times, and that often
only certains lines are needed. Using grep is much faster that doing
this with perl. I also replaced a loop with a perl hash and moved an
expression which always evaluates to the same value out of a loop.
But read the commit mails of the patches, they explain a lot in
detail.

I also tried to optimize some perl regex, but my tests didn't gain a
notable improvement, so if left the regex unchanged.
-- 
regards Thomas

Bug#871835: speed up for debootstrap

[Tiago Ilieve]

Hi Thomas,

Do you mind to share how you identified those slower parts of the
code? Have you used a profiler or a similar tool?

I must say that I'm very impressed by your patches. :-)

Regards,
Tiago.

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://de.linkedin.com/in/myhro
Berlin, Deutschland

Bug#871835: speed up for debootstrap

[Cyril Brulebois]

Hi Thomas,

Thomas Lange  (2017-08-12):
> Package: debootstrap
> Severity: normal
> Tags: patch
> 
> I've wrote some patches, which speed up deboostrap a lot.
> Changes are only made to /usr/share/debootstrap/functions, mostly in
> the perl part, but also the number of lines that are proceeded by
> perl are reduced via egrep.
> 
> Here are some time measures on a 4-core machine, the target directory
> is inside a tmpfs and a local mirror in the same subnet is used.
> 
> old version 1.0.89
> real1m26.940s
> user1m24.180s
> sys 0m2.420s
> 
> 
> including the patches
> real0m39.481s
> user0m50.444s
> sys 0m2.748s
> 
> 
> If you have any questions concerning the patches, just ask me.

I'm very excited about those patches, feel free to catch me during the
evening for a little talk about those.


KiBi.


signature.asc
Description: Digital signature

Bug#871835: speed up for debootstrap

[Thomas Lange]

Package: debootstrap
Severity: normal
Tags: patch

I've wrote some patches, which speed up deboostrap a lot.
Changes are only made to /usr/share/debootstrap/functions, mostly in
the perl part, but also the number of lines that are proceeded by
perl are reduced via egrep.

Here are some time measures on a 4-core machine, the target directory
is inside a tmpfs and a local mirror in the same subnet is used.

old version 1.0.89
real1m26.940s
user1m24.180s
sys 0m2.420s


including the patches
real0m39.481s
user0m50.444s
sys 0m2.748s


If you have any questions concerning the patches, just ask me.



0001-optimize-code-make-debootstrap-faster.patch
Description: Binary data


0002-use-hash-instead-of-grep-for-each-line-we-read.patch
Description: Binary data


0003-we-can-use-elsif-because-the-line-can-t-match-both-p.patch
Description: Binary data


0004-There-s-no-need-to-print-the-dependencies-for-each-p.patch
Description: Binary data


0005-we-can-get-the-size-by-using-stat-instead-of-reading.patch
Description: Binary data


0006-do-not-read-all-lines-of-the-Packages-file-into-the-.patch
Description: Binary data

-- 
regards Thomas