Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder

2019-10-13 Thread Nicholas D Steeves
Hi Daniel and everyone reading this,

Daniel  writes:

> I am addressing another case, the one you have not separated partitions 
> for /, /home and swap.
>

Len and Daniel, WRT swap, hibernation is useful when you need to
preserve the state of applications that aren't aware of a desktop
session manager, eg: half-finished work in a terminal.  As I see it the
primary advantage to a swap file is the ability to expand it after a RAM
upgrade. eg: that a reboot using a livecd to resize partitions is not
required.  Granted, expanding a swap file is only necessary if one cares
about hibernation...

> As a matter of fact if the installer is able to recognize the home 
> folder havingĀ  /home separated in another partition is not necessary 
> anymore. The advantage respect having the /home separated, specifically 
> for a desktop use are noteworthy.
>

The key point is "is able to recognize [safely and reliably]", and that
requires someone who values the feature to work on debian-installer
(not fun), and also maintain it (not fun).

> If the installer, instead of creating /, /home and swap, creates just / 
> and a swap file and if is able to reinstall itself without overwriting 
> the home folder I think is a huge improvement. As a matter of fact if 
> you reinstall Debian, even with /home in another partition, there is not 
> any assisted aid that explain you how to properly setup the /home 
> partition. Having the system partitioned is already a setup for advanced 
> user.
>

As Len mentioned, Debian users tend to believe that it's a better use of
time to learn how to fix things if one is going to track
testing/sid/experimental than hitting the panic button and losing state.
My mother (who lives 3500km away, runs Debian stable, and installed it
herself eight years ago) doesn't need this feature.  She's not an
intermediate or advanced user and is currently running buster.  She also
has root on her laptop, so has the power to render it unusable.

Daniel, would you please take a look at Bug #941627 (ITP: grub-btrfs --
provides grub entries for btrfs snapshots (boot environments/restore
points)?  I think that this is a feature that would solve the situation
where one ran a testing/sid/experimental upgrade at a time when one
didn't have time to do the work required to fix the brokenness.

Here's how it will work:

1. Install to btrfs (after my MR is merged this will automatically
create @rootfs subvolume, eg: special directory kind of like a pseudo
logical volume).  Reboot.
2. Run a one-line command and add one line to fstab to create a @home
subvolume--this is necessary to exclude /home from the snapshots.  I can
write a beginner friendly helper script if necessary.
3. Take a snapshot before running a dangerous upgrade (easy one-line
command).  Eventually this may be automatic (eg: something like
apt-btrfs-snapshot)
4. If the upgrade produces a broken state the user doesn't have time to
fix, simply boot into a known-good copy of / using the grub menu to
select the correct entry.  If the top-most one isn't good, reboot and
try with the second top-most one until a good one is found.  After
confirming all is well, rename the @rootfs subvolume and create a new
read-write snapshot named @rootfs based on the current boot environment.
This step is only necessary if you want to reboot into the old copy of
the rootfs automatically.  You also get to keep the
@rootfs-does-not-work copy.
5. The logical progression of this feature is to create a snapshot,
dist-upgrade the snapshot, test it (without rebooting), and if
everything looks good then mark it as the default boot environment.

Eventually there will be a GUI!

While wiping and reinstalling may be the best other OSs can aspire to,
Debian can, and will, do better.  I hope you'll agree the project I'm
working on will solve the root issue you're reporting, and that you'll
agree it's a more elegant and time-efficient solution :-)

In the meantime, to remove everything but /home and reinstall without
reformatting you can reboot into a Debian rescue system or using the
Debian Live image, mount your volume, use the solution I provided in my
initial reply (p.s. I consider that a risky approach), then follow:

  https://www.debian.org/releases/buster/amd64/apds03.en.html
  https://wiki.debian.org/Debootstrap


Sorry for the belated reply, I've been swamped with work.
Cheers,
Nicholas


signature.asc
Description: PGP signature


Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder

2019-10-02 Thread Lennart Sorensen
On Mon, Sep 30, 2019 at 12:51:33PM -0400, Daniel wrote:
> Dear Lennart,
> 
> I hope that when one opens a "whishlist bug" at least there is a chance to
> have a confrontation.
> 
> The main point I want to address is when you do a "smart installation" it is
> supposed to perform a clean installation hence the only folder that must to
> be untouched is "/home". The same concept when you have "/" and "/home" in
> separated partitions and you perform a clean installation. I think that is
> pretty trivial, the smart parts are:
> 
> * the installer is able to check for a previous Debian installation before
> to begin the process;
> 
> * and in case it founds a previous installation, the installer, is able to
> perform a fresh installation without overwriting the "/home" folder.

Well I believe you have the option to not format a mountpoint during
the install already, so at least that part should be pretty easy.

> I can confirm that ElementaryOS and POP!_OS, that share the same installer,
> can do that.

Well hopefully someone will try to contribute that then.  I suspect the
main thing is finding someone that wants to implement it and do the work
to add it and maintain it.

> Last point I want touch is about the swap partition. With the SSD and the OS
> able to boot in a bunch of seconds the hibernation doesn't make any sense
> today. For example I have 16GB of ram, based on the standard rules I should
> use at least 1.5x of the ram if not the double. It means that I should use
> 32GB just to hibernate my session, no way... With the SSD disks the lesser
> you write on the disk the better, I put just 2GB of swap-file and
> "swappiness" at 1 and the swap is never used and I didn't waste 30GB of
> space.

Only advantage to huibernation is not having to close all the things
you are working on and opening them again after the next boot.  I do
find hibernation takes too long with a lot of ram and hence never do it
myself. :)

> To conclude I think I elaborated everything clearly, I see a lot of benefits
> and improvement with the suggestions I gave to Debian, I also think that are
> pretty trivial to implement. I don't want introduce a Windows behavior of
> "reinstall when it broken", but back to time when I hadn't a fast internet
> connection it was faster download the full ISO and performing a fresh
> installation rather than doing a "dist-upgrade".

I remember upgrades over dial up.  Still did not make me want to go
download full iso images elsewhere.  It could do it while I slept.
Things have gotten a lot bigger since then though.  I have seen people
keep a subset mirror of Debian on a USB drive that they would update with
rsync once in a while at work, and bring home to use for upgrades where
the connection was slow.  Still in place upgrades of course, not using
the installer.

> The bottom line is with a smart installer you don't need to separate your
> disk(s) in partitions but you can throw everything in "/" including the
> "swap" as swap-file that you can modify freely based on your needs (if you
> can't live without hibernation[1]). There is also a dynamic swap manager
> available on Debian as well: https://github.com/Tookmund/Swapspace

-- 
Len Sorensen



Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder

2019-09-30 Thread Daniel

Dear Lennart,

I hope that when one opens a "whishlist bug" at least there is a chance 
to have a confrontation.


The main point I want to address is when you do a "smart installation" 
it is supposed to perform a clean installation hence the only folder 
that must to be untouched is "/home". The same concept when you have "/" 
and "/home" in separated partitions and you perform a clean 
installation. I think that is pretty trivial, the smart parts are:


* the installer is able to check for a previous Debian installation 
before to begin the process;


* and in case it founds a previous installation, the installer, is able 
to perform a fresh installation without overwriting the "/home" folder.


I can confirm that ElementaryOS and POP!_OS, that share the same 
installer, can do that.



Last point I want touch is about the swap partition. With the SSD and 
the OS able to boot in a bunch of seconds the hibernation doesn't make 
any sense today. For example I have 16GB of ram, based on the standard 
rules I should use at least 1.5x of the ram if not the double. It means 
that I should use 32GB just to hibernate my session, no way... With the 
SSD disks the lesser you write on the disk the better, I put just 2GB of 
swap-file and "swappiness" at 1 and the swap is never used and I didn't 
waste 30GB of space.


To conclude I think I elaborated everything clearly, I see a lot of 
benefits and improvement with the suggestions I gave to Debian, I also 
think that are pretty trivial to implement. I don't want introduce a 
Windows behavior of "reinstall when it broken", but back to time when I 
hadn't a fast internet connection it was faster download the full ISO 
and performing a fresh installation rather than doing a "dist-upgrade".


The bottom line is with a smart installer you don't need to separate 
your disk(s) in partitions but you can throw everything in "/" including 
the "swap" as swap-file that you can modify freely based on your needs 
(if you can't live without hibernation[1]). There is also a dynamic swap 
manager available on Debian as well: https://github.com/Tookmund/Swapspace



My best,

Daniel

[1] It needs some tuning to work: 
https://wiki.archlinux.org/index.php/Power_management/Suspend_and_hibernate#Hibernation_into_swap_file



On 9/30/19 9:49 AM, Lennart Sorensen wrote:

On Sat, Sep 28, 2019 at 11:27:29PM -0400, Daniel wrote:

Hi Nicholas,

thanks for your reply, I really appreciated your constructive approach.

I use Debian since 2007 and I did a lot of installation, I personally use a
FrankenDebian (testing with pinning toward SID and Experimental) however
when I install Debian on other machines I install definitely the current
stable available. I have been performing exclusively desktop installations
and while I consider the best option separating /home recently I found
myself not able to get the right balance between "/", "/home" and "swap".
The default "/" assigned is often too small while sometimes I wasted
gigabyte never used. The "swap" with the amount of ram available today is
always more accessory and with the SSD disk the trend is to reduce its use
the most. Eventually I stopped to create a "swap" partition in favor of a
"swap-file" (like Raspian e.g.); hence I also stopped to create "/" and
"/home" but just "/" and still as LVM; at this point you don't have anymore
issue with the space and if you need you can add all the disks you want
because it is still a LVM partition.

Now the case I am figuring out is the one you didn't separe "/" and "/home"
(however the installer is still creating "swap") but you need to reinstall
Debian because you screwed it up for some reason. Now a smart installer
before to start everything takes its time to check the disk and discovers
that you have, along a crypted disk and a LVM group, also a previous version
of Debian hence check the users and it asks you if you want keep all the
users, just one, etc... and then it reinstalls the system and recovers the
setting from the user(s) you selected, without creating a FrankenDebian but
just a fresh and **smart** installation.

This leads in my opinion in creating a further voice for the Debian install:
**the desktop installation**; Standard and Advanced are eventually too
generic and do not target properly the desktop cases. If the D-I was
properly able to read LUKS and LVM during the installation time, and if was
also able to perform a smart installation as described in the paragraph
above, a Desktop installation should be:

1. Create an encrypted partition by default (LUKS + LVM);

I rarely do that, but I can see why some people want it.


2. install everything in / ;

I do tend to prefer that for most setups myself.


3. not create a "swap partition" but a swap-file.

My understanding is that suspend to disk works much easier with a swap
partition still, but my information could be out of date on that.
And of course swap smaller than ram makes suspend to disk not possible.


I also add that:

4. 

Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder

2019-09-30 Thread Lennart Sorensen
On Sat, Sep 28, 2019 at 11:27:29PM -0400, Daniel wrote:
> Hi Nicholas,
> 
> thanks for your reply, I really appreciated your constructive approach.
> 
> I use Debian since 2007 and I did a lot of installation, I personally use a
> FrankenDebian (testing with pinning toward SID and Experimental) however
> when I install Debian on other machines I install definitely the current
> stable available. I have been performing exclusively desktop installations
> and while I consider the best option separating /home recently I found
> myself not able to get the right balance between "/", "/home" and "swap".
> The default "/" assigned is often too small while sometimes I wasted
> gigabyte never used. The "swap" with the amount of ram available today is
> always more accessory and with the SSD disk the trend is to reduce its use
> the most. Eventually I stopped to create a "swap" partition in favor of a
> "swap-file" (like Raspian e.g.); hence I also stopped to create "/" and
> "/home" but just "/" and still as LVM; at this point you don't have anymore
> issue with the space and if you need you can add all the disks you want
> because it is still a LVM partition.
> 
> Now the case I am figuring out is the one you didn't separe "/" and "/home"
> (however the installer is still creating "swap") but you need to reinstall
> Debian because you screwed it up for some reason. Now a smart installer
> before to start everything takes its time to check the disk and discovers
> that you have, along a crypted disk and a LVM group, also a previous version
> of Debian hence check the users and it asks you if you want keep all the
> users, just one, etc... and then it reinstalls the system and recovers the
> setting from the user(s) you selected, without creating a FrankenDebian but
> just a fresh and **smart** installation.
> 
> This leads in my opinion in creating a further voice for the Debian install:
> **the desktop installation**; Standard and Advanced are eventually too
> generic and do not target properly the desktop cases. If the D-I was
> properly able to read LUKS and LVM during the installation time, and if was
> also able to perform a smart installation as described in the paragraph
> above, a Desktop installation should be:
> 
> 1. Create an encrypted partition by default (LUKS + LVM);

I rarely do that, but I can see why some people want it.

> 2. install everything in / ;

I do tend to prefer that for most setups myself.

> 3. not create a "swap partition" but a swap-file.

My understanding is that suspend to disk works much easier with a swap
partition still, but my information could be out of date on that.
And of course swap smaller than ram makes suspend to disk not possible.

> I also add that:
> 
> 4. should deactivate root user by default, which is now considering a best
> practice;

Not sure I agree it is considered best practices.  A lot of distributions
do it, but not all.  I do prefer root login to work from the console if
I have to fix something.

> 5. should deactivate the source repos and asking to activate the "contrib"
> and "non-free" repos (like in Advanced Mode).
> 
> 
> I don't see any complicated tasks to achieve, others Linux distro already
> started to move in this direction while other *nix operative systems already
> do that since a long time.

Other distributions (Certainly the case for redhat based stuff in the
past) had to do it since they didn't have a working in place upgrade.
That rather makes it required that the installer can do an upgrade and
detect existing settings.  Debian seems to have always aimed for an in
place upgrade that worked, so the installer really only had the purpose
of the initial install.  It's one of the things that made me switch to
Debian over 20 years ago.  I have never had to do a reinstall of a Debian
system except on a machine that lost the disk and I didn't have a backup
of it (nothing important was kept on that system).  I really should
have replaced that other disk in the RAID1 within a reasonable amount
of time. :)

> The only issues I see here are the resistance to the changes and the fact
> that actually the D-I has some issue to recognize the encrypted partitions
> and if you want reinstall Debian you can't preserve any of the partitions
> you want because it will consider the encrypted disks as blanks.

Collecting all those settings does not sound like a trivial job, and based
on the normal use case of a Debian install, I sure don't see the value
in it.  How do you even decide which settings should be preserved and
which should not?  What if one of the settings is what broke your system?
If you screw up the system, go fix it.  You will learn something from it.
Blowing away the system and installing it again means you learn nothing,
waste a bunch of time, and will likely do it again in the future.  I have
certainly broken my installs over the years and had to fix it, but it has
always been possible.  Running unstable and experimental stuff at times

Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder

2019-09-28 Thread Daniel

Hi Nicholas,

thanks for your reply, I really appreciated your constructive approach.

I use Debian since 2007 and I did a lot of installation, I personally 
use a FrankenDebian (testing with pinning toward SID and Experimental) 
however when I install Debian on other machines I install definitely the 
current stable available. I have been performing exclusively desktop 
installations and while I consider the best option separating /home 
recently I found myself not able to get the right balance between "/", 
"/home" and "swap". The default "/" assigned is often too small while 
sometimes I wasted gigabyte never used. The "swap" with the amount of 
ram available today is always more accessory and with the SSD disk the 
trend is to reduce its use the most. Eventually I stopped to create a 
"swap" partition in favor of a "swap-file" (like Raspian e.g.); hence I 
also stopped to create "/" and "/home" but just "/" and still as LVM; at 
this point you don't have anymore issue with the space and if you need 
you can add all the disks you want because it is still a LVM partition.


Now the case I am figuring out is the one you didn't separe "/" and 
"/home" (however the installer is still creating "swap") but you need to 
reinstall Debian because you screwed it up for some reason. Now a smart 
installer before to start everything takes its time to check the disk 
and discovers that you have, along a crypted disk and a LVM group, also 
a previous version of Debian hence check the users and it asks you if 
you want keep all the users, just one, etc... and then it reinstalls the 
system and recovers the setting from the user(s) you selected, without 
creating a FrankenDebian but just a fresh and **smart** installation.


This leads in my opinion in creating a further voice for the Debian 
install: **the desktop installation**; Standard and Advanced are 
eventually too generic and do not target properly the desktop cases. If 
the D-I was properly able to read LUKS and LVM during the installation 
time, and if was also able to perform a smart installation as described 
in the paragraph above, a Desktop installation should be:


1. Create an encrypted partition by default (LUKS + LVM);

2. install everything in / ;

3. not create a "swap partition" but a swap-file.

I also add that:

4. should deactivate root user by default, which is now considering a 
best practice;


5. should deactivate the source repos and asking to activate the 
"contrib" and "non-free" repos (like in Advanced Mode).



I don't see any complicated tasks to achieve, others Linux distro 
already started to move in this direction while other *nix operative 
systems already do that since a long time.


The only issues I see here are the resistance to the changes and the 
fact that actually the D-I has some issue to recognize the encrypted 
partitions and if you want reinstall Debian you can't preserve any of 
the partitions you want because it will consider the encrypted disks as 
blanks.



Best regards,

Daniel




On 9/28/19 12:01 AM, Nicholas D Steeves wrote:

Geert Stappers  writes:


On Fri, Sep 27, 2019 at 05:19:06PM -0400, Daniel wrote:

Holger Wansing wrote:

The debian-installer supports similar use case via the "separate
partition for /home" approach.

to reinstall Debian on top of itself without overwriting the home partition.

Yes, that is what Holger is telling.

I think Daniel is requesting an option that does something like this:

   find /install-target -maxdepth 1 | grep -v 'home\|lost+found' | xargs rm -rf

Maybe this way isn't robust enough, but active mounts shouldn't have
their mount points removed, because

   rm: cannot remove '/install-target/foo': Device or resource busy

BTW, Daniel, you can decruft your system with "apt purge --autoremove
foo", which also deletes config in /etc and will notify you if any files
remain in /var.  One of the greatest strengths of Debian is that unlike
other operating systems, smooth upgrades between stable versions are
taken seriously...gravely seriously...so one never needs to reinstall.
The only things that I've seen that have ever required action are
packages that needed manual configuration updates in /etc (equally
solvable by apt purge), and obsolete/broken configuration in /home/user
(not solved if this feature request is implemented).  What problem is
this feature request intended to solve?  FrankenDebian?

   https://wiki.debian.org/DontBreakDebian


Cheers,
Nicholas

P.S. apt install installation-birthday  :-)




Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder

2019-09-27 Thread Daniel
I am addressing another case, the one you have not separated partitions 
for /, /home and swap.


As a matter of fact if the installer is able to recognize the home 
folder havingĀ  /home separated in another partition is not necessary 
anymore. The advantage respect having the /home separated, specifically 
for a desktop use are noteworthy.


If the installer, instead of creating /, /home and swap, creates just / 
and a swap file and if is able to reinstall itself without overwriting 
the home folder I think is a huge improvement. As a matter of fact if 
you reinstall Debian, even with /home in another partition, there is not 
any assisted aid that explain you how to properly setup the /home 
partition. Having the system partitioned is already a setup for advanced 
user.


Thanks,

D.

On 9/27/19 6:45 PM, Geert Stappers wrote:

On Fri, Sep 27, 2019 at 05:19:06PM -0400, Daniel wrote:

Holger Wansing wrote:

The debian-installer supports similar use case via the "separate
partition for /home" approach.

to reinstall Debian on top of itself without overwriting the home partition.

Yes, that is what Holger is telling.


Groeten
Geert Stappers




Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder

2019-09-27 Thread Nicholas D Steeves
Geert Stappers  writes:

> On Fri, Sep 27, 2019 at 05:19:06PM -0400, Daniel wrote:
>> Holger Wansing wrote:
>> > The debian-installer supports similar use case via the "separate
>> > partition for /home" approach.
>> to reinstall Debian on top of itself without overwriting the home partition.
>
> Yes, that is what Holger is telling.

I think Daniel is requesting an option that does something like this:

  find /install-target -maxdepth 1 | grep -v 'home\|lost+found' | xargs rm -rf

Maybe this way isn't robust enough, but active mounts shouldn't have
their mount points removed, because

  rm: cannot remove '/install-target/foo': Device or resource busy

BTW, Daniel, you can decruft your system with "apt purge --autoremove
foo", which also deletes config in /etc and will notify you if any files
remain in /var.  One of the greatest strengths of Debian is that unlike
other operating systems, smooth upgrades between stable versions are
taken seriously...gravely seriously...so one never needs to reinstall.
The only things that I've seen that have ever required action are
packages that needed manual configuration updates in /etc (equally
solvable by apt purge), and obsolete/broken configuration in /home/user
(not solved if this feature request is implemented).  What problem is
this feature request intended to solve?  FrankenDebian?

  https://wiki.debian.org/DontBreakDebian


Cheers,
Nicholas

P.S. apt install installation-birthday  :-)


signature.asc
Description: PGP signature


Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder

2019-09-27 Thread Geert Stappers
On Fri, Sep 27, 2019 at 05:19:06PM -0400, Daniel wrote:
> Holger Wansing wrote:
> > The debian-installer supports similar use case via the "separate
> > partition for /home" approach.
> to reinstall Debian on top of itself without overwriting the home partition.

Yes, that is what Holger is telling.


Groeten
Geert Stappers
-- 
Leven en laten leven



Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder

2019-09-27 Thread Daniel

Dear All,

In this case I suggest hence to expand the use cases and to evaluate the 
feature to reinstall Debian on top of itself without overwriting the 
home partition. While it doesn't seem a complicated achievement, it 
brings instead a great advantage for the end user and improves greatly 
the desktop experience, for new users as well for the power users.


My regards,

Daniel