Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[ValdikSS]

Hi.

I'm writing here to share my experience with eatmydata in the debian-installer.

I was installing Debian 10 on a USB stick.  The PC has a USB 2.0 port
and the stick's writing speed is about 4.6 MiB/s.  I did not measure
the installation time precisely, but I'm sure it took at least 6 hours
to complete
with an LXQt desktop and an Apache server.

I installed manually eatmydata, libeatmydata1 and eatmydata-udeb
on the installer's filesystem, by extracting the contents of the archives
to their right locations.  I do not remember using anna-install to install
the udeb, I don't know if it would make a difference.
Also, I used the normal DVD-1 ISO image, without eatmydata in it.

I would recommend you to try to install Debian using virtual machine, 
installing it from the patched ISO files: 
https://bitbucket.org/ValdikSS/debian-iso-fastinstall/src


Already patched ISOs are available here: 
ftp://serv.valdikss.org.ru/Downloads/Linux%20ISOs/Debian%20ISO%20FastInstall/





OpenPGP_signature
Description: OpenPGP digital signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Cmdte Alpha Tigre Z]

Hi.

I'm writing here to share my experience with eatmydata in the debian-installer.

I was installing Debian 10 on a USB stick.  The PC has a USB 2.0 port
and the stick's writing speed is about 4.6 MiB/s.  I did not measure
the installation time precisely, but I'm sure it took at least 6 hours
to complete
with an LXQt desktop and an Apache server.

I installed manually eatmydata, libeatmydata1 and eatmydata-udeb
on the installer's filesystem, by extracting the contents of the archives
to their right locations.  I do not remember using anna-install to install
the udeb, I don't know if it would make a difference.
Also, I used the normal DVD-1 ISO image, without eatmydata in it.

I wanted debootstrap to be called with eatmydata from the installer's menu,
so I renamed debootstrap as "debootstrap1" and put along it a script called
"debootstrap" which does 'eatmydata debootstrap "$@"'.  I made the patch
described at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700633#50
but debootstrap complained that it cannot find eatmydata in the ISO, so I
installed it by hand, no apt, I don't know if this changed the
expected behavior.
Also, I don't know if using eatmydata explicitly with the script and later using
the scripts included with eatmydata-udeb cause some conflict and make it
not work.

I saw at the fourth virtual console that libeatmydata1.so was not being found
by the installer in the target directory tree, it was because I forgot
to make the
symbolic links pointing to the real library.  I made it while the
installation was
already running and I stopped seeing the messages come out in the syslog;
I suppose it was detected later, but I don't know if it worked as it
should since
I did this at the middle of the installation process, not before.

Anyway, perhaps I did something wrong, these are the things I remember
I did not do as specified in the above message.  But if everything
should have worked right, then eatmydata doesn't work as I expected
in my USB stick, or it is just too slow and it can not install faster.

What do you say?

Have a good day.

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Charles Curley]

On Fri, 16 Apr 2021 08:49:34 +0100
Ian Campbell  wrote:

> > 
> > Please do not confuse "ignore" with "terribly understaffed".  
> 
> I wonder if there is anything which a 3rd party could be found to do
> via the Freexian project thing[0] which would reduce the overall
> burden and free up some of the limited time contributors (of which I
> am long lapsed :-() have? e.g. automate some time consuming manual
> work, setup automated testing infra, that sort of thing?

Thanks for bringing this up.

I am a long time Debian user (Linux since 1993, Debian only for the
last decade or so). I have had little idea how things work in the
Debian community and am just beginning to explore it.

Along with Freexian, perhaps the occasional Call for Volunteers to get
specific things done would shake loose some volunteers? These could go
out on the various user-level mailing lists such as debian-user.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Ian Campbell]

(just the list, other cc'd dropped)

On Mon, 2021-04-12 at 19:35 +0200, Samuel Thibault wrote:
> Petter Reinholdtsen, le lun. 12 avril 2021 07:34:38 +0200, a ecrit:
> > Sad to hear the patch has been ignored for several years.
> 
> Please do not confuse "ignore" with "terribly understaffed".

I wonder if there is anything which a 3rd party could be found to do
via the Freexian project thing[0] which would reduce the overall burden
and free up some of the limited time contributors (of which I am long
lapsed :-() have? e.g. automate some time consuming manual work, setup
automated testing infra, that sort of thing?

Ian.

[0] https://salsa.debian.org/freexian-team/project-funding
 

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Petter Reinholdtsen]

[Samuel Thibault]
> In my understanding, "ignore" carries a negative meaning, as if d-b
> people didn't *want* to have a look at the proposed patches.

OK.  Personally, I ignore a lot of things due to lack of capacity, and
while it can of course be negative for those being ignored, it has a
positive effect on me avoiding overload.  To me the difference of not
noticing and ignoring is the choice one make about what to look at.  I
assume the understaffed team decide which issues to look at and which to
ignore, even if the decision process is is "pick random issue, leave the
rest* or "pick issue that look interesting, leave rest for others* or
something else. :)

Anyway, I am happy to report that action point 4 was just fixed in git
according to https://bugs.debian.org/986772 >.  This should make
it simple to fix action point 3 too, enabling eatmydata-udeb in the
default installation, before the release of Bullseye.

As for action points 1 and 2, patching debootstrap and d-i to use new
option, I seriously doubt it can be implemented and tested quickly
enough, but would love to be wrong.

Perhaps all that is needed is more positive test reports for the patch
in https://bugs.debian.org/700633 >. :)

-- 
Happy hacking
Petter Reinholdtsen

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Holger Levsen]

On Wed, Apr 14, 2021 at 08:38:00AM +0200, Petter Reinholdtsen wrote:
> [...]  Please let me know via
> https://bugs.debian.org/986772 > if you agree or disagree with
> that change, and I can apply it if no-one objects by the end of the week.

for those not following that bug, this has been fixed in git now and as
thus will be applied to future builds.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁   holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄

In Germany we don‘t say „Happy Valentine‘s Day, I love you“, we say „ich werde
diesen vom Markt kreierten, konsumorientierten Trend des Kapitalismus nicht
unterstützen,“ and I think that’s beautiful. (Hazel Brugger)


signature.asc
Description: PGP signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Samuel Thibault]

Petter Reinholdtsen, le mer. 14 avril 2021 08:38:00 +0200, a ecrit:
> [Samuel Thibault]
> > Petter Reinholdtsen, le lun. 12 avril 2021 07:34:38 +0200, a ecrit:
> >> Sad to hear the patch has been ignored for several years.
> >
> > Please do not confuse "ignore" with "terribly understaffed".
> 
> In my experience, "terribly understaffed" can lead to many things being
> ignored, and I this fund ignored to be the correct term for it.

In my understanding, "ignore" carries a negative meaning, as if d-b
people didn't *want* to have a look at the proposed patches.

https://www.dictionary.com/browse/ignore

«
to refrain from noticing or recognizing:

Law. (of a grand jury) to reject (a bill of indictment), as on the grounds of 
insufficient evidence.
»

being understaffed doesn't imply refraining or rejecting, but simply not
having the time to even just look at it.

Not having time to delve more.
Samuel

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Petter Reinholdtsen]

Thank you for the CC.  I am not subscribed to debian-boot@ and it would
have take a while before I discovered your reply on the web view.

[Samuel Thibault]
> Petter Reinholdtsen, le lun. 12 avril 2021 07:34:38 +0200, a ecrit:
>> Sad to hear the patch has been ignored for several years.
>
> Please do not confuse "ignore" with "terribly understaffed".

In my experience, "terribly understaffed" can lead to many things being
ignored, and I this fund ignored to be the correct term for it.  The
staff level is useful as an explanation but do not really change the
effect, which is that changes for the better do not get applied.

I could apply the debian-cd change myself, if everyone agree that it
should be applied, but have not touched debian-cd for so long it seemed
safest to only publish the patch as a start.  Please let me know via
https://bugs.debian.org/986772 > if you agree or disagree with
that change, and I can apply it if no-one objects by the end of the week.

I am not sadly qualified at the moment to look at debootstrap, and
unlikely to find time to reload that code base into my head any time
soon.  Sorry about that.  Looked at the ubuntu patch for debootstrap, as
it came from a @ubuntu user, but unfortunately they are not carrying it
yet, so it has not been testet there.

-- 
Happy hacking
Petter Reinholdtsen

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Samuel Thibault]

Petter Reinholdtsen, le lun. 12 avril 2021 07:34:38 +0200, a ecrit:
> Sad to hear the patch has been ignored for several years.

Please do not confuse "ignore" with "terribly understaffed".

Samuel

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Petter Reinholdtsen]

[ValdikSS]
> 1. To speed up base system install (the first installation step),
> debootstrap should be improved to use libeatmydata for internal dpkg
> call. There's a patch for it, which adds '--include-early' option, but
> it haven't got attention from debootstrap developers:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700633#50
> My debian-iso-fastinstall script crudely patches debootstrap script
> with sed — not a production-ready solution.

Sad to hear the patch has been ignored for several years.

> 2. d-i should run debootstrap as:
> eatmydata debootstrap --include-early=eatmydata ...

Can only be fixed after the patch in BTS #700633 is fixed.

> 3. d-i should call 'anna-install eatmydata-udeb' on the start of
> installation. I guess there's some kind of udeb list in the installer,
> eatmydata-udeb package should be added to it. 
> My debian-iso-fastinstall script includes this command as a preseed
> kernel configuration argument, which rewrites anything in a real preseed
> exec command, so not a production-ready as well.

This of course can be done immediately by the d-i team.  I suspect the
easiest way to do it is by modifying the package priority in the
archive, not by adding a 'anna-install' line in the code.  If so, the
priority might have to be set by the ftpmasters.  Not sure how the
interaction between the ftpmasters and the d-i team is regarding udeb
archive priorities, but assume the d-i team have the last say here.

> 4. ISO image should include eatmydata and libeatmydata packages in
> /pool directory. Right now only eatmydata-udeb is included.

This need to be done in the debian-cd package, so I created the patch in
https://bugs.debian.org/986772 > to try to make it happen.  This
will make sure the required debs are available on the first "CD" for
those that want to use eatmydata-udeb for tasksel, but not affect the
debootstrap part.  Luckiliy, the debootstrap part is often shorter than
the tasksel part of the installation.

So there are three groups of developers that need to move for all of
this to happen, the people maintaining debootstrap, separate set of
people maintaining debian-cd and the separate set of people maintaining
debian-installer.  While there is some overlap, given the time frame of
the request for quicker installations in Debian, it remains to be seen
if any of these see enabling eatmydata-udeb as a priority.

-- 
Happy Hacking
Petter Reinholdtsen

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[ValdikSS]

I have questions, isn't eatmydata-udeb enough to get it working for d-i?
It's not enough: it's only activates eatmydata, but for some reason does 
not depend on it. I don't know why is it so.


Please also note that eatmydata-udeb works only on a later installation 
stages and does nothing for base system installation.




Does it have other dependencies on non-udeb packages or files
that are not in the initrd of the installer?
I thought that udeb packages were made to work as standalones,
with less dependencies.  If eatmydata-udeb is not enough to do
something with it, then it should be a bug.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Cmdte Alpha Tigre Z]

Ok, thank you for the additional information.

El vie, 9 abr 2021 a las 1:20, ValdikSS () escribió:

> 4. ISO image should include eatmydata and libeatmydata packages in /pool 
> directory. Right now only eatmydata-udeb is included.

I have questions, isn't eatmydata-udeb enough to get it working for d-i?
Does it have other dependencies on non-udeb packages or files
that are not in the initrd of the installer?
I thought that udeb packages were made to work as standalones,
with less dependencies.  If eatmydata-udeb is not enough to do
something with it, then it should be a bug.

--
Time zone: GMT-4

PS: I forgot to send this message to the list.  Sorry.

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Holger Levsen (2021-04-09 10:34:53)
> On Fri, Apr 09, 2021 at 08:20:42AM +0300, ValdikSS wrote:
> > 1. To speed up base system install (the first installation step),
> > debootstrap should be improved to use libeatmydata for internal dpkg call. 
> > [...]
> 
> or maybe mmdebstrap could be used instead? AFAIK it doesn't use libeatmydata
> but achieves the same by other means.
> 
> I don't know how feasable creating an udeb for it (and it depends/recommends)
> is, cc:ing the author for input.

the problem is, that mmdebstrap by its nature requires Perl and apt and neither
have a udeb as of today. There is a bug about this wishlist feature:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=909724

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Holger Levsen]

On Fri, Apr 09, 2021 at 08:20:42AM +0300, ValdikSS wrote:
> 1. To speed up base system install (the first installation step),
> debootstrap should be improved to use libeatmydata for internal dpkg call. 
> [...]

or maybe mmdebstrap could be used instead? AFAIK it doesn't use libeatmydata
but achieves the same by other means.

I don't know how feasable creating an udeb for it (and it depends/recommends)
is, cc:ing the author for input.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁   holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄

:wq


signature.asc
Description: PGP signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[ValdikSS]

As this proposal sounds like a very good idea, and since
it doesn't look to difficult to be done now, before the release of
Debian 11, the Debian-Boot team should do this.

What do we need to do to push this proposal to d-i for Debian 11?


1. To speed up base system install (the first installation step), 
debootstrap should be improved to use libeatmydata for internal dpkg 
call. There's a patch for it, which adds '--include-early' option, but 
it haven't got attention from debootstrap developers: 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700633#50
My debian-iso-fastinstall script crudely patches debootstrap script with 
sed — not a production-ready solution.


2. d-i should run debootstrap as:
eatmydata debootstrap --include-early=eatmydata ...

3. d-i should call 'anna-install eatmydata-udeb' on the start of 
installation. I guess there's some kind of udeb list in the installer, 
eatmydata-udeb package should be added to it.
My debian-iso-fastinstall script includes this command as a preseed 
kernel configuration argument, which rewrites anything in a real preseed 
exec command, so not a production-ready as well.


4. ISO image should include eatmydata and libeatmydata packages in /pool 
directory. Right now only eatmydata-udeb is included.





OpenPGP_signature
Description: OpenPGP digital signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Cmdte Alpha Tigre Z]

2021-04-07 4:51 GMT-04:00, john doe :
> This move should be documented somewhere and maybe emit a warning to
> that effect before starting the installer:
>
> "Incase of failure during the installation, it is recommended to restart
> from scratch."

That warning sounds good, but perhaps it should be there
whether you are using eatmydata or not.
I would not like to finish an interrupted partial installation too.

> Could d-i verify that the installation was successfull and tell it to
> the user?

As others have said, doing that thing that eatmydata disables
just at the end before d-i reports
the end of the installation procedure,
there should not be more risk using eatmydata than not using it.

As this proposal sounds like a very good idea, and since
it doesn't look to difficult to be done now, before the release of
Debian 11, the Debian-Boot team should do this.

What do we need to do to push this proposal to d-i for Debian 11?
--
Santiago Pinto

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Philip Hands]

[resent -- to list this time]

john doe  writes:

> This move should be documented somewhere and maybe emit a warning to
> that effect before starting the installer:
>
> "Incase of failure during the installation, it is recommended to restart
> from scratch."
>
> Could d-i verify that the installation was successfull and tell it to
> the user?

That warning is valid (although perhaps redundant) whether using
eatmydata or not.

Trying to continue a partial install that was interrupted by a power
failure is either an amusing experiment (I suspect I've done it at some
point on that basis) or either foolhardy or pointless depending on where
the failure happened (well, unless the install was actually done).

I guess the "Instalation Complete" step of the installer could include
running a sync before announcing itself, but beyond that if you think
additional testing is required simply because something called
"eatmydata" was in use at some point, then you've not understood what it
does.

Cheers, Phil.

P.S. I am fully in favour of this change, and hope to have some time to
help with the effort shortly.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Philip Hands]

john doe  writes:

> Please send through the list, so other can react to your answer/feedback.

That mail was intended to be sent to the list, but I upgraded notmuch
recently and seem to have had it revert to it's default keybindings.

Sorry about that.

However, you should be cautious about forwarding mails sent privately.
It might be that someone else would feel that you'd somehow infringed
their privacy, so it's better to reply privately and let the sender
resend their mail (that also keeps the threads in better shape, so I'll
send the mail again so it's from me if people want to reply)

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[john doe]

Please send through the list, so other can react to your answer/feedback.

On 4/7/2021 11:32 AM, Philip Hands wrote:

john doe  writes:


This move should be documented somewhere and maybe emit a warning to
that effect before starting the installer:

"Incase of failure during the installation, it is recommended to restart
from scratch."

Could d-i verify that the installation was successfull and tell it to
the user?


That warning is valid (although perhaps redundant) whether using
eatmydata or not.

Trying to continue a partial install that was interrupted by a power
failure is either an amusing experiment (I suspect I've done it at some
point on that basis) or either foolhardy or pointless depending on where
the failure happened (well, unless the install was actually done).

I guess the "Instalation Complete" step of the installer could include
running a sync before announcing itself, but beyond that if you think
additional testing is required simply because something called
"eatmydata" was in use at some point, then you've not understood what it
does.

Cheers, Phil.

P.S. I am fully in favour of this change, and hope to have some time to
help with the effort shortly.




--
John Doe

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Samuel Thibault]

john doe, le mer. 07 avril 2021 10:51:08 +0200, a ecrit:
> This move should be documented somewhere and maybe emit a warning to
> that effect before starting the installer:
> 
> "Incase of failure during the installation, it is recommended to restart
> from scratch."

That has always been so. Even if dpkg is in a coherent state, it's
really not clear how restarting from there can lead to anything good.

> Could d-i verify that the installation was successfull and tell it to
> the user?

That's the final message before the reboot yes.
We can probably just add an explicit sync before emitting it.

Samuel

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[john doe]

On 4/6/2021 11:46 PM, Lennart Sorensen wrote:

On Tue, Apr 06, 2021 at 05:35:19PM -0400, Cmdte Alpha Tigre Z wrote:

Pardon my ignorance.
I could not resist to answer to this proposal.

I read this page: https://www.flamingspork.com/projects/libeatmydata/

It looks like it is not good idea to use it for critical information.

However, your results show that it would be relevant to include
such package into the first ISO, if it is not so big, of course.
It sounds like a good idea to speed up the instalation process
for some cases.

But, I would not enable it by default.  If the package really
behaves as its name suggests,
I would not risk every user
of Debian to have a faulty installation.  What would happen
if someone wants to install Debian an suddenly, the installer
eats some data it shouldn't have.

Even if it does not access wrong places, in the worst case
you could have installed an ill OS and don't notice
it will someday fail, and not gracefully.

My humble opinion is that it should be available to use,
but not enabled by default.


The only time eatmydata does any harm is if the system looses power or
resets during the install since the data isn't constantly flushed to disk
to maintain a consistent state.  During an install, there is nothing of
value on the system yet, so doing everything as quickly as possible and
then when everything is done, then you issue a sync command to ensure
everything is flushed to disk saves a ton of time with no risk at all
(in fact since the install takes less time, the changes of a power
interruption happening during the install is lowered).



This move should be documented somewhere and maybe emit a warning to
that effect before starting the installer:

"Incase of failure during the installation, it is recommended to restart
from scratch."


Could d-i verify that the installation was successfull and tell it to
the user?

--
John Doe

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Cmdte Alpha Tigre Z]

El mar, 6 abr 2021 a las 18:45, Samuel Thibault
() escribió:
>
> Cmdte Alpha Tigre Z, le mar. 06 avril 2021 18:16:49 -0400, a ecrit:
> > What would happen if you were installing Debian to dual-boot
> > with another OS?
>
> That is not a concern, only the partition getting installed is at risk.

Ok.  So in case of a power cut, the other partitions remain unaffected.

> > What would happen if you were repartitioning the disk
> > with some other stuff in it? ...
>
> That, however, has to run without eatmydata.
>
> But AIUI the idea is to make only debootstrap and the apt-based install
> use eatmydata.

Hmm, if that's the case, then I do think it is a good idea
to include and activate by default eatmydata.  debian-installer must,
however, be selective when using eatmydata; for example, it should
be disabled when using partman, to avoid cases like that presented above.
If it works only when doing the Debian installation per-se, which is
the most slow process, then there is nothing to lose.

Two days ago, I was trying to install Debian on a USB stick.
I think it took around 4 hours.  It would be better to install it
a lot faster without having any real added risk.

-- 
Time zone: GMT-4

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Samuel Thibault]

Cmdte Alpha Tigre Z, le mar. 06 avril 2021 18:16:49 -0400, a ecrit:
> What would happen if you were installing Debian to dual-boot
> with another OS?

That is not a concern, only the partition getting installed is at risk.

> What would happen if you were repartitioning the disk
> with some other stuff in it? ...

That, however, has to run without eatmydata.

But AIUI the idea is to make only debootstrap and the apt-based install
use eatmydata.

Samuel

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Cmdte Alpha Tigre Z]

Emm.  Sorry.  The mail was accidentally
sent twice.

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Cmdte Alpha Tigre Z]

2021-04-06 17:41 GMT-04:00, Samuel Thibault :
> Indeed. But a fresh install is not critical information. If you lost
> power during installation you'll want to restart over anyway.
>
> Its name is there for people to indeed not think that it's a magic wand
> that makes everything faster. It makes it faster at the expense of not
> protecting it from power loss. But here the half-installed system is
> moot anyway, we do not care that the dpkg database is coherent, we'll
> want to start over anyway.

It do makes sense to start over the installation if there were a
power failure.

2021-04-06 17:46 GMT-04:00, Lennart Sorensen  The only time eatmydata does any harm is if the system looses power or
> resets during the install since the data isn't constantly flushed to disk
> to maintain a consistent state.

Thanks for this clarification,
I know understand better the risk.

> During an install, there is nothing of
> value on the system yet, so doing everything as quickly as possible and
> then when everything is done, then you issue a sync command to ensure
> everything is flushed to disk saves a ton of time with no risk at all
> (in fact since the install takes less time, the changes of a power
> interruption happening during the install is lowered).
>
> In no way does eatmydata make it possible for the data of the resulting
> install to be corrupt.  As long as the filesystem is cleanly unmounted
> or flushed before you reset, you are fine.  That should already happen
> by the fact the install does a clean reboot at the end.
>
> Using it on a normal system is a different story since anything you modify
> while using it could be lost in case the system is reset unexpectedly,
> but since the install has no user data, there is nothing to risk.
>
> So when the page says to not use when you care about the data, that
> is correct.  But a fresh install is entirely made of stuff you don't
> care about, until it is completely done, then you care.  Using it for
> running testsuites where everything is just temporary data also makes
> sense to speed that up.  If you are editing something real, don't use it.
>
> --
> Len Sorensen
>

Ok.  Thanks Mr. Thibault and Mr. Sorensen.  I now understand
what is this proposal about.

Just some little questions, I'm still in doubt:
What would happen if you were installing Debian to dual-boot
with another OS?
What would happen if you were repartitioning the disk
with some other stuff in it? ...
and suddenly, your PC had a power loss.

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Cmdte Alpha Tigre Z]

2021-04-06 17:41 GMT-04:00, Samuel Thibault :
> Indeed. But a fresh install is not critical information. If you lost
> power during installation you'll want to restart over anyway.
>
> Its name is there for people to indeed not think that it's a magic wand
> that makes everything faster. It makes it faster at the expense of not
> protecting it from power loss. But here the half-installed system is
> moot anyway, we do not care that the dpkg database is coherent, we'll
> want to start over anyway.

It do makes sense to start over the installation if there were a
power failure.

2021-04-06 17:46 GMT-04:00, Lennart Sorensen  The only time eatmydata does any harm is if the system looses power or
> resets during the install since the data isn't constantly flushed to disk
> to maintain a consistent state.

Thanks for this clarification,
I know understand better the risk.

> During an install, there is nothing of
> value on the system yet, so doing everything as quickly as possible and
> then when everything is done, then you issue a sync command to ensure
> everything is flushed to disk saves a ton of time with no risk at all
> (in fact since the install takes less time, the changes of a power
> interruption happening during the install is lowered).
>
> In no way does eatmydata make it possible for the data of the resulting
> install to be corrupt.  As long as the filesystem is cleanly unmounted
> or flushed before you reset, you are fine.  That should already happen
> by the fact the install does a clean reboot at the end.
>
> Using it on a normal system is a different story since anything you modify
> while using it could be lost in case the system is reset unexpectedly,
> but since the install has no user data, there is nothing to risk.
>
> So when the page says to not use when you care about the data, that
> is correct.  But a fresh install is entirely made of stuff you don't
> care about, until it is completely done, then you care.  Using it for
> running testsuites where everything is just temporary data also makes
> sense to speed that up.  If you are editing something real, don't use it.
>
> --
> Len Sorensen
>

Ok.  Thanks Mr. Thibault and Mr. Sorensen.  I now understand
what is this proposal about.

Just some little questions, I'm still in doubt:
What would happen if you were installing Debian to dual-boot
with another OS?
What would happen if you were repartitioning the disk
with some other stuff in it? ...
and suddenly, your PC had a power loss.

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Lennart Sorensen]

On Tue, Apr 06, 2021 at 05:35:19PM -0400, Cmdte Alpha Tigre Z wrote:
> Pardon my ignorance.
> I could not resist to answer to this proposal.
> 
> I read this page: https://www.flamingspork.com/projects/libeatmydata/
> 
> It looks like it is not good idea to use it for critical information.
> 
> However, your results show that it would be relevant to include
> such package into the first ISO, if it is not so big, of course.
> It sounds like a good idea to speed up the instalation process
> for some cases.
> 
> But, I would not enable it by default.  If the package really
> behaves as its name suggests,
> I would not risk every user
> of Debian to have a faulty installation.  What would happen
> if someone wants to install Debian an suddenly, the installer
> eats some data it shouldn't have.
> 
> Even if it does not access wrong places, in the worst case
> you could have installed an ill OS and don't notice
> it will someday fail, and not gracefully.
> 
> My humble opinion is that it should be available to use,
> but not enabled by default.

The only time eatmydata does any harm is if the system looses power or
resets during the install since the data isn't constantly flushed to disk
to maintain a consistent state.  During an install, there is nothing of
value on the system yet, so doing everything as quickly as possible and
then when everything is done, then you issue a sync command to ensure
everything is flushed to disk saves a ton of time with no risk at all
(in fact since the install takes less time, the changes of a power
interruption happening during the install is lowered).

In no way does eatmydata make it possible for the data of the resulting
install to be corrupt.  As long as the filesystem is cleanly unmounted
or flushed before you reset, you are fine.  That should already happen
by the fact the install does a clean reboot at the end.

Using it on a normal system is a different story since anything you modify
while using it could be lost in case the system is reset unexpectedly,
but since the install has no user data, there is nothing to risk.

So when the page says to not use when you care about the data, that
is correct.  But a fresh install is entirely made of stuff you don't
care about, until it is completely done, then you care.  Using it for
running testsuites where everything is just temporary data also makes
sense to speed that up.  If you are editing something real, don't use it.

-- 
Len Sorensen

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Samuel Thibault]

Cmdte Alpha Tigre Z, le mar. 06 avril 2021 17:35:19 -0400, a ecrit:
> I read this page: https://www.flamingspork.com/projects/libeatmydata/
> 
> It looks like it is not good idea to use it for critical information.

Indeed. But a fresh install is not critical information. If you lost
power during installation you'll want to restart over anyway.

> But, I would not enable it by default.  If the package really
> behaves as its name suggests,

Its name is there for people to indeed not think that it's a magic wand
that makes everything faster. It makes it faster at the expense of not
protecting it from power loss. But here the half-installed system is
moot anyway, we do not care that the dpkg database is coherent, we'll
want to start over anyway.

Samuel

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Cmdte Alpha Tigre Z]

2021-04-06 14:24 GMT-04:00, ValdikSS :
> People on #debian-boot IRC told me to contact dpkg team. Guillem Jover,
> one of dpkg maintainer, posted the following reply in dpkg
> --force-unsafe-io still calls fsync()
>  bug from
> 2014:
>
>> If someone can perform
>> more tests showing otherwise, please feel free to reopen and I'll
>> consider extending or adding a new option for a full-unsafe-io mode,
>> but as it stands, I don't think that makese any sense, when what you
>> might want is to just use eatmydata or similar.
>
> I've contacted him but unfortunately received no reply.
> I'd want to improve installation speed, but I don't know how to proceed
> further, and since Debian Bullseye is already at freeze state, I'm
> afraid we may end with slow installation for the next 2 years. I'd
> greatly appreciate any help.
>
>
> On 14.03.2021 03:18, ValdikSS wrote:
>>
>> Debian Buster installation from DVD1 ISO file takes enormous amount of
>> time in a VM without write cache and on bare metal with older (a bit
>> wore out) HDDs due to excessive fsync() calls.
>> For example, installation from debian-10.6.0-amd64-DVD-1.iso in a VM
>> without internet access took 1 hour, 44 minutes, 30 seconds. On a bare
>> metal: more than 40 minutes (I was too impatient, didn't wait it to
>> finish and powered off the machine).
>>
>> I found out that eatmydata-udeb package, which was implemented to
>> speed up the installation process, is included into ISO images (in
>> both CD and DVD), but can't be used because eatmydata and
>> libeatmydata1 packages are missing. Enabling eatmydata-udeb does not
>> have any effect in this case.
>>
>> The issues with eatmydata-udeb in my opinion are:
>>
>>  1. eatmydata-udeb should be enabled by default, to speed up the
>> installation process. It should not require to be enabled from
>> preseed file or bootloader cmdline. It's pointless to use fsync()
>> in installation process, the user won't attempt to fix partially
>> installed system due to power loss, it only slows it down.
>>  2. eatmydata and libeatmydata1 packages should present in ISO images
>> (/pool directory) to make eatmydata-udeb possible to use without
>> the internet.
>>  3. eatmydata-udeb should also inject libeatmydata to the base system
>> installation process, which is performed by debootstrap. Base
>> system installation is also quite slow, not as slow as the main
>> system due to much less package count, but still takes much more
>> time than it should.
>> See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700633
>> 
>>
>>
>> I've prepared an automatic patcher script which could be found here:
>> https://bitbucket.org/ValdikSS/debian-iso-fastinstall/
>> It adds eatmydata and libeatmydata1 packages into ISO image, patches
>> debootstrap to use libeatmydata, and activates eatmydata-udeb (patches
>> boot cmdline).
>>
>> With this script, patched debian-10.6.0-amd64-DVD-1.iso ISO file which
>> previously took 1 hour, 44 minutes, 30 seconds to install, now
>> installs in 10 minutes 37 seconds.
>> Patched netinstall image, when run without the internet access,
>> installs the whole mini distro with base utilities in less than 3
>> minutes.
>>
>> Petter Reinholdtsen, the author of eatmydata-udeb, asked me to post my
>> findings here for public discussion. What do you think?
>>
>>
>

Pardon my ignorance.
I could not resist to answer to this proposal.

I read this page: https://www.flamingspork.com/projects/libeatmydata/

It looks like it is not good idea to use it for critical information.

However, your results show that it would be relevant to include
such package into the first ISO, if it is not so big, of course.
It sounds like a good idea to speed up the instalation process
for some cases.

But, I would not enable it by default.  If the package really
behaves as its name suggests,
I would not risk every user
of Debian to have a faulty installation.  What would happen
if someone wants to install Debian an suddenly, the installer
eats some data it shouldn't have.

Even if it does not access wrong places, in the worst case
you could have installed an ill OS and don't notice
it will someday fail, and not gracefully.

My humble opinion is that it should be available to use,
but not enabled by default.

Have a good day.

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[ValdikSS]

People on #debian-boot IRC told me to contact dpkg team. Guillem Jover, 
one of dpkg maintainer, posted the following reply in dpkg 
--force-unsafe-io still calls fsync() 
 bug from 
2014:



If someone can perform
more tests showing otherwise, please feel free to reopen and I'll
consider extending or adding a new option for a full-unsafe-io mode,
but as it stands, I don't think that makese any sense, when what you
might want is to just use eatmydata or similar.


I've contacted him but unfortunately received no reply.
I'd want to improve installation speed, but I don't know how to proceed 
further, and since Debian Bullseye is already at freeze state, I'm 
afraid we may end with slow installation for the next 2 years. I'd 
greatly appreciate any help.



On 14.03.2021 03:18, ValdikSS wrote:


Debian Buster installation from DVD1 ISO file takes enormous amount of 
time in a VM without write cache and on bare metal with older (a bit 
wore out) HDDs due to excessive fsync() calls.
For example, installation from debian-10.6.0-amd64-DVD-1.iso in a VM 
without internet access took 1 hour, 44 minutes, 30 seconds. On a bare 
metal: more than 40 minutes (I was too impatient, didn't wait it to 
finish and powered off the machine).


I found out that eatmydata-udeb package, which was implemented to 
speed up the installation process, is included into ISO images (in 
both CD and DVD), but can't be used because eatmydata and 
libeatmydata1 packages are missing. Enabling eatmydata-udeb does not 
have any effect in this case.


The issues with eatmydata-udeb in my opinion are:

 1. eatmydata-udeb should be enabled by default, to speed up the
installation process. It should not require to be enabled from
preseed file or bootloader cmdline. It's pointless to use fsync()
in installation process, the user won't attempt to fix partially
installed system due to power loss, it only slows it down.
 2. eatmydata and libeatmydata1 packages should present in ISO images
(/pool directory) to make eatmydata-udeb possible to use without
the internet.
 3. eatmydata-udeb should also inject libeatmydata to the base system
installation process, which is performed by debootstrap. Base
system installation is also quite slow, not as slow as the main
system due to much less package count, but still takes much more
time than it should.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700633



I've prepared an automatic patcher script which could be found here: 
https://bitbucket.org/ValdikSS/debian-iso-fastinstall/
It adds eatmydata and libeatmydata1 packages into ISO image, patches 
debootstrap to use libeatmydata, and activates eatmydata-udeb (patches 
boot cmdline).


With this script, patched debian-10.6.0-amd64-DVD-1.iso ISO file which 
previously took 1 hour, 44 minutes, 30 seconds to install, now 
installs in 10 minutes 37 seconds.
Patched netinstall image, when run without the internet access, 
installs the whole mini distro with base utilities in less than 3 minutes.


Petter Reinholdtsen, the author of eatmydata-udeb, asked me to post my 
findings here for public discussion. What do you think?





OpenPGP_signature
Description: OpenPGP digital signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Samuel Thibault]

Charles Curley, le sam. 13 mars 2021 19:03:17 -0700, a ecrit:
> On Sun, 14 Mar 2021 03:18:12 +0300
> ValdikSS  wrote:
> 
> >  1. eatmydata-udeb should be enabled by default, to speed up the
> > installation process. It should not require to be enabled from
> > preseed file or bootloader cmdline. It's pointless to use fsync()
> > in installation process, the user won't attempt to fix partially
> > installed system due to power loss, it only slows it down.
> 
> I am concerned with low memory installations. I suggest that under low
> memory conditions eatmydata-udeb should be optional.

?

- If you are concerned with the space used by the .udeb itself, under
  very low memory conditions the lowmem package already asks the user
  to choose which udebs to enable. TBH, "lowmem" nowadays doesn't care
  about a few dozen KiB :)

- If you are concerned with possibly large page cache usage, well it's
  up to the Linux kernel to flush when it feels obliged to, and at this
  point the swap is enabled already anyway.

Samuel

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[ValdikSS]

I am concerned with low memory installations. I suggest that under low
memory conditions eatmydata-udeb should be optional.

Do you have any data that would help a low memory installer decide
whether to use it or not?


libeatmydata is so small that enabling it does not have any measurable 
effect to memory consumption. The library is 13 KB in size on disk and 
adds maybe +8KB (two pages) of overall RAM usage (it's a shared library).


P.S. please answer me directly, I'm not subscribed to the list.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[Charles Curley]

On Sun, 14 Mar 2021 03:18:12 +0300
ValdikSS  wrote:

>  1. eatmydata-udeb should be enabled by default, to speed up the
> installation process. It should not require to be enabled from
> preseed file or bootloader cmdline. It's pointless to use fsync()
> in installation process, the user won't attempt to fix partially
> installed system due to power loss, it only slows it down.

I am concerned with low memory installations. I suggest that under low
memory conditions eatmydata-udeb should be optional.

Do you have any data that would help a low memory installer decide
whether to use it or not?

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/


pgpVjkRnkRCcw.pgp
Description: OpenPGP digital signature

Speed up installation: activate eatmydata-udeb by default and include eatmydata package in /pool

[ValdikSS]

Debian Buster installation from DVD1 ISO file takes enormous amount of 
time in a VM without write cache and on bare metal with older (a bit 
wore out) HDDs due to excessive fsync() calls.
For example, installation from debian-10.6.0-amd64-DVD-1.iso in a VM 
without internet access took 1 hour, 44 minutes, 30 seconds. On a bare 
metal: more than 40 minutes (I was too impatient, didn't wait it to 
finish and powered off the machine).


I found out that eatmydata-udeb package, which was implemented to speed 
up the installation process, is included into ISO images (in both CD and 
DVD), but can't be used because eatmydata and libeatmydata1 packages are 
missing. Enabling eatmydata-udeb does not have any effect in this case.


The issues with eatmydata-udeb in my opinion are:

1. eatmydata-udeb should be enabled by default, to speed up the
   installation process. It should not require to be enabled from
   preseed file or bootloader cmdline. It's pointless to use fsync() in
   installation process, the user won't attempt to fix partially
   installed system due to power loss, it only slows it down.
2. eatmydata and libeatmydata1 packages should present in ISO images
   (/pool directory) to make eatmydata-udeb possible to use without the
   internet.
3. eatmydata-udeb should also inject libeatmydata to the base system
   installation process, which is performed by debootstrap. Base system
   installation is also quite slow, not as slow as the main system due
   to much less package count, but still takes much more time than it
   should.
   See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700633
   


I've prepared an automatic patcher script which could be found here: 
https://bitbucket.org/ValdikSS/debian-iso-fastinstall/
It adds eatmydata and libeatmydata1 packages into ISO image, patches 
debootstrap to use libeatmydata, and activates eatmydata-udeb (patches 
boot cmdline).


With this script, patched debian-10.6.0-amd64-DVD-1.iso ISO file which 
previously took 1 hour, 44 minutes, 30 seconds to install, now installs 
in 10 minutes 37 seconds.
Patched netinstall image, when run without the internet access, installs 
the whole mini distro with base utilities in less than 3 minutes.


Petter Reinholdtsen, the author of eatmydata-udeb, asked me to post my 
findings here for public discussion. What do you think?





OpenPGP_signature
Description: OpenPGP digital signature