Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder
Hi Daniel and everyone reading this, Daniel writes: > I am addressing another case, the one you have not separated partitions > for /, /home and swap. > Len and Daniel, WRT swap, hibernation is useful when you need to preserve the state of applications that aren't aware of a desktop session manager, eg: half-finished work in a terminal. As I see it the primary advantage to a swap file is the ability to expand it after a RAM upgrade. eg: that a reboot using a livecd to resize partitions is not required. Granted, expanding a swap file is only necessary if one cares about hibernation... > As a matter of fact if the installer is able to recognize the home > folder having /home separated in another partition is not necessary > anymore. The advantage respect having the /home separated, specifically > for a desktop use are noteworthy. > The key point is "is able to recognize [safely and reliably]", and that requires someone who values the feature to work on debian-installer (not fun), and also maintain it (not fun). > If the installer, instead of creating /, /home and swap, creates just / > and a swap file and if is able to reinstall itself without overwriting > the home folder I think is a huge improvement. As a matter of fact if > you reinstall Debian, even with /home in another partition, there is not > any assisted aid that explain you how to properly setup the /home > partition. Having the system partitioned is already a setup for advanced > user. > As Len mentioned, Debian users tend to believe that it's a better use of time to learn how to fix things if one is going to track testing/sid/experimental than hitting the panic button and losing state. My mother (who lives 3500km away, runs Debian stable, and installed it herself eight years ago) doesn't need this feature. She's not an intermediate or advanced user and is currently running buster. She also has root on her laptop, so has the power to render it unusable. Daniel, would you please take a look at Bug #941627 (ITP: grub-btrfs -- provides grub entries for btrfs snapshots (boot environments/restore points)? I think that this is a feature that would solve the situation where one ran a testing/sid/experimental upgrade at a time when one didn't have time to do the work required to fix the brokenness. Here's how it will work: 1. Install to btrfs (after my MR is merged this will automatically create @rootfs subvolume, eg: special directory kind of like a pseudo logical volume). Reboot. 2. Run a one-line command and add one line to fstab to create a @home subvolume--this is necessary to exclude /home from the snapshots. I can write a beginner friendly helper script if necessary. 3. Take a snapshot before running a dangerous upgrade (easy one-line command). Eventually this may be automatic (eg: something like apt-btrfs-snapshot) 4. If the upgrade produces a broken state the user doesn't have time to fix, simply boot into a known-good copy of / using the grub menu to select the correct entry. If the top-most one isn't good, reboot and try with the second top-most one until a good one is found. After confirming all is well, rename the @rootfs subvolume and create a new read-write snapshot named @rootfs based on the current boot environment. This step is only necessary if you want to reboot into the old copy of the rootfs automatically. You also get to keep the @rootfs-does-not-work copy. 5. The logical progression of this feature is to create a snapshot, dist-upgrade the snapshot, test it (without rebooting), and if everything looks good then mark it as the default boot environment. Eventually there will be a GUI! While wiping and reinstalling may be the best other OSs can aspire to, Debian can, and will, do better. I hope you'll agree the project I'm working on will solve the root issue you're reporting, and that you'll agree it's a more elegant and time-efficient solution :-) In the meantime, to remove everything but /home and reinstall without reformatting you can reboot into a Debian rescue system or using the Debian Live image, mount your volume, use the solution I provided in my initial reply (p.s. I consider that a risky approach), then follow: https://www.debian.org/releases/buster/amd64/apds03.en.html https://wiki.debian.org/Debootstrap Sorry for the belated reply, I've been swamped with work. Cheers, Nicholas signature.asc Description: PGP signature
Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder
On Mon, Sep 30, 2019 at 12:51:33PM -0400, Daniel wrote: > Dear Lennart, > > I hope that when one opens a "whishlist bug" at least there is a chance to > have a confrontation. > > The main point I want to address is when you do a "smart installation" it is > supposed to perform a clean installation hence the only folder that must to > be untouched is "/home". The same concept when you have "/" and "/home" in > separated partitions and you perform a clean installation. I think that is > pretty trivial, the smart parts are: > > * the installer is able to check for a previous Debian installation before > to begin the process; > > * and in case it founds a previous installation, the installer, is able to > perform a fresh installation without overwriting the "/home" folder. Well I believe you have the option to not format a mountpoint during the install already, so at least that part should be pretty easy. > I can confirm that ElementaryOS and POP!_OS, that share the same installer, > can do that. Well hopefully someone will try to contribute that then. I suspect the main thing is finding someone that wants to implement it and do the work to add it and maintain it. > Last point I want touch is about the swap partition. With the SSD and the OS > able to boot in a bunch of seconds the hibernation doesn't make any sense > today. For example I have 16GB of ram, based on the standard rules I should > use at least 1.5x of the ram if not the double. It means that I should use > 32GB just to hibernate my session, no way... With the SSD disks the lesser > you write on the disk the better, I put just 2GB of swap-file and > "swappiness" at 1 and the swap is never used and I didn't waste 30GB of > space. Only advantage to huibernation is not having to close all the things you are working on and opening them again after the next boot. I do find hibernation takes too long with a lot of ram and hence never do it myself. :) > To conclude I think I elaborated everything clearly, I see a lot of benefits > and improvement with the suggestions I gave to Debian, I also think that are > pretty trivial to implement. I don't want introduce a Windows behavior of > "reinstall when it broken", but back to time when I hadn't a fast internet > connection it was faster download the full ISO and performing a fresh > installation rather than doing a "dist-upgrade". I remember upgrades over dial up. Still did not make me want to go download full iso images elsewhere. It could do it while I slept. Things have gotten a lot bigger since then though. I have seen people keep a subset mirror of Debian on a USB drive that they would update with rsync once in a while at work, and bring home to use for upgrades where the connection was slow. Still in place upgrades of course, not using the installer. > The bottom line is with a smart installer you don't need to separate your > disk(s) in partitions but you can throw everything in "/" including the > "swap" as swap-file that you can modify freely based on your needs (if you > can't live without hibernation[1]). There is also a dynamic swap manager > available on Debian as well: https://github.com/Tookmund/Swapspace -- Len Sorensen
Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder
Dear Lennart, I hope that when one opens a "whishlist bug" at least there is a chance to have a confrontation. The main point I want to address is when you do a "smart installation" it is supposed to perform a clean installation hence the only folder that must to be untouched is "/home". The same concept when you have "/" and "/home" in separated partitions and you perform a clean installation. I think that is pretty trivial, the smart parts are: * the installer is able to check for a previous Debian installation before to begin the process; * and in case it founds a previous installation, the installer, is able to perform a fresh installation without overwriting the "/home" folder. I can confirm that ElementaryOS and POP!_OS, that share the same installer, can do that. Last point I want touch is about the swap partition. With the SSD and the OS able to boot in a bunch of seconds the hibernation doesn't make any sense today. For example I have 16GB of ram, based on the standard rules I should use at least 1.5x of the ram if not the double. It means that I should use 32GB just to hibernate my session, no way... With the SSD disks the lesser you write on the disk the better, I put just 2GB of swap-file and "swappiness" at 1 and the swap is never used and I didn't waste 30GB of space. To conclude I think I elaborated everything clearly, I see a lot of benefits and improvement with the suggestions I gave to Debian, I also think that are pretty trivial to implement. I don't want introduce a Windows behavior of "reinstall when it broken", but back to time when I hadn't a fast internet connection it was faster download the full ISO and performing a fresh installation rather than doing a "dist-upgrade". The bottom line is with a smart installer you don't need to separate your disk(s) in partitions but you can throw everything in "/" including the "swap" as swap-file that you can modify freely based on your needs (if you can't live without hibernation[1]). There is also a dynamic swap manager available on Debian as well: https://github.com/Tookmund/Swapspace My best, Daniel [1] It needs some tuning to work: https://wiki.archlinux.org/index.php/Power_management/Suspend_and_hibernate#Hibernation_into_swap_file On 9/30/19 9:49 AM, Lennart Sorensen wrote: On Sat, Sep 28, 2019 at 11:27:29PM -0400, Daniel wrote: Hi Nicholas, thanks for your reply, I really appreciated your constructive approach. I use Debian since 2007 and I did a lot of installation, I personally use a FrankenDebian (testing with pinning toward SID and Experimental) however when I install Debian on other machines I install definitely the current stable available. I have been performing exclusively desktop installations and while I consider the best option separating /home recently I found myself not able to get the right balance between "/", "/home" and "swap". The default "/" assigned is often too small while sometimes I wasted gigabyte never used. The "swap" with the amount of ram available today is always more accessory and with the SSD disk the trend is to reduce its use the most. Eventually I stopped to create a "swap" partition in favor of a "swap-file" (like Raspian e.g.); hence I also stopped to create "/" and "/home" but just "/" and still as LVM; at this point you don't have anymore issue with the space and if you need you can add all the disks you want because it is still a LVM partition. Now the case I am figuring out is the one you didn't separe "/" and "/home" (however the installer is still creating "swap") but you need to reinstall Debian because you screwed it up for some reason. Now a smart installer before to start everything takes its time to check the disk and discovers that you have, along a crypted disk and a LVM group, also a previous version of Debian hence check the users and it asks you if you want keep all the users, just one, etc... and then it reinstalls the system and recovers the setting from the user(s) you selected, without creating a FrankenDebian but just a fresh and **smart** installation. This leads in my opinion in creating a further voice for the Debian install: **the desktop installation**; Standard and Advanced are eventually too generic and do not target properly the desktop cases. If the D-I was properly able to read LUKS and LVM during the installation time, and if was also able to perform a smart installation as described in the paragraph above, a Desktop installation should be: 1. Create an encrypted partition by default (LUKS + LVM); I rarely do that, but I can see why some people want it. 2. install everything in / ; I do tend to prefer that for most setups myself. 3. not create a "swap partition" but a swap-file. My understanding is that suspend to disk works much easier with a swap partition still, but my information could be out of date on that. And of course swap smaller than ram makes suspend to disk not possible. I also add that: 4.
Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder
On Sat, Sep 28, 2019 at 11:27:29PM -0400, Daniel wrote: > Hi Nicholas, > > thanks for your reply, I really appreciated your constructive approach. > > I use Debian since 2007 and I did a lot of installation, I personally use a > FrankenDebian (testing with pinning toward SID and Experimental) however > when I install Debian on other machines I install definitely the current > stable available. I have been performing exclusively desktop installations > and while I consider the best option separating /home recently I found > myself not able to get the right balance between "/", "/home" and "swap". > The default "/" assigned is often too small while sometimes I wasted > gigabyte never used. The "swap" with the amount of ram available today is > always more accessory and with the SSD disk the trend is to reduce its use > the most. Eventually I stopped to create a "swap" partition in favor of a > "swap-file" (like Raspian e.g.); hence I also stopped to create "/" and > "/home" but just "/" and still as LVM; at this point you don't have anymore > issue with the space and if you need you can add all the disks you want > because it is still a LVM partition. > > Now the case I am figuring out is the one you didn't separe "/" and "/home" > (however the installer is still creating "swap") but you need to reinstall > Debian because you screwed it up for some reason. Now a smart installer > before to start everything takes its time to check the disk and discovers > that you have, along a crypted disk and a LVM group, also a previous version > of Debian hence check the users and it asks you if you want keep all the > users, just one, etc... and then it reinstalls the system and recovers the > setting from the user(s) you selected, without creating a FrankenDebian but > just a fresh and **smart** installation. > > This leads in my opinion in creating a further voice for the Debian install: > **the desktop installation**; Standard and Advanced are eventually too > generic and do not target properly the desktop cases. If the D-I was > properly able to read LUKS and LVM during the installation time, and if was > also able to perform a smart installation as described in the paragraph > above, a Desktop installation should be: > > 1. Create an encrypted partition by default (LUKS + LVM); I rarely do that, but I can see why some people want it. > 2. install everything in / ; I do tend to prefer that for most setups myself. > 3. not create a "swap partition" but a swap-file. My understanding is that suspend to disk works much easier with a swap partition still, but my information could be out of date on that. And of course swap smaller than ram makes suspend to disk not possible. > I also add that: > > 4. should deactivate root user by default, which is now considering a best > practice; Not sure I agree it is considered best practices. A lot of distributions do it, but not all. I do prefer root login to work from the console if I have to fix something. > 5. should deactivate the source repos and asking to activate the "contrib" > and "non-free" repos (like in Advanced Mode). > > > I don't see any complicated tasks to achieve, others Linux distro already > started to move in this direction while other *nix operative systems already > do that since a long time. Other distributions (Certainly the case for redhat based stuff in the past) had to do it since they didn't have a working in place upgrade. That rather makes it required that the installer can do an upgrade and detect existing settings. Debian seems to have always aimed for an in place upgrade that worked, so the installer really only had the purpose of the initial install. It's one of the things that made me switch to Debian over 20 years ago. I have never had to do a reinstall of a Debian system except on a machine that lost the disk and I didn't have a backup of it (nothing important was kept on that system). I really should have replaced that other disk in the RAID1 within a reasonable amount of time. :) > The only issues I see here are the resistance to the changes and the fact > that actually the D-I has some issue to recognize the encrypted partitions > and if you want reinstall Debian you can't preserve any of the partitions > you want because it will consider the encrypted disks as blanks. Collecting all those settings does not sound like a trivial job, and based on the normal use case of a Debian install, I sure don't see the value in it. How do you even decide which settings should be preserved and which should not? What if one of the settings is what broke your system? If you screw up the system, go fix it. You will learn something from it. Blowing away the system and installing it again means you learn nothing, waste a bunch of time, and will likely do it again in the future. I have certainly broken my installs over the years and had to fix it, but it has always been possible. Running unstable and experimental stuff at times
Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder
Hi Nicholas, thanks for your reply, I really appreciated your constructive approach. I use Debian since 2007 and I did a lot of installation, I personally use a FrankenDebian (testing with pinning toward SID and Experimental) however when I install Debian on other machines I install definitely the current stable available. I have been performing exclusively desktop installations and while I consider the best option separating /home recently I found myself not able to get the right balance between "/", "/home" and "swap". The default "/" assigned is often too small while sometimes I wasted gigabyte never used. The "swap" with the amount of ram available today is always more accessory and with the SSD disk the trend is to reduce its use the most. Eventually I stopped to create a "swap" partition in favor of a "swap-file" (like Raspian e.g.); hence I also stopped to create "/" and "/home" but just "/" and still as LVM; at this point you don't have anymore issue with the space and if you need you can add all the disks you want because it is still a LVM partition. Now the case I am figuring out is the one you didn't separe "/" and "/home" (however the installer is still creating "swap") but you need to reinstall Debian because you screwed it up for some reason. Now a smart installer before to start everything takes its time to check the disk and discovers that you have, along a crypted disk and a LVM group, also a previous version of Debian hence check the users and it asks you if you want keep all the users, just one, etc... and then it reinstalls the system and recovers the setting from the user(s) you selected, without creating a FrankenDebian but just a fresh and **smart** installation. This leads in my opinion in creating a further voice for the Debian install: **the desktop installation**; Standard and Advanced are eventually too generic and do not target properly the desktop cases. If the D-I was properly able to read LUKS and LVM during the installation time, and if was also able to perform a smart installation as described in the paragraph above, a Desktop installation should be: 1. Create an encrypted partition by default (LUKS + LVM); 2. install everything in / ; 3. not create a "swap partition" but a swap-file. I also add that: 4. should deactivate root user by default, which is now considering a best practice; 5. should deactivate the source repos and asking to activate the "contrib" and "non-free" repos (like in Advanced Mode). I don't see any complicated tasks to achieve, others Linux distro already started to move in this direction while other *nix operative systems already do that since a long time. The only issues I see here are the resistance to the changes and the fact that actually the D-I has some issue to recognize the encrypted partitions and if you want reinstall Debian you can't preserve any of the partitions you want because it will consider the encrypted disks as blanks. Best regards, Daniel On 9/28/19 12:01 AM, Nicholas D Steeves wrote: Geert Stappers writes: On Fri, Sep 27, 2019 at 05:19:06PM -0400, Daniel wrote: Holger Wansing wrote: The debian-installer supports similar use case via the "separate partition for /home" approach. to reinstall Debian on top of itself without overwriting the home partition. Yes, that is what Holger is telling. I think Daniel is requesting an option that does something like this: find /install-target -maxdepth 1 | grep -v 'home\|lost+found' | xargs rm -rf Maybe this way isn't robust enough, but active mounts shouldn't have their mount points removed, because rm: cannot remove '/install-target/foo': Device or resource busy BTW, Daniel, you can decruft your system with "apt purge --autoremove foo", which also deletes config in /etc and will notify you if any files remain in /var. One of the greatest strengths of Debian is that unlike other operating systems, smooth upgrades between stable versions are taken seriously...gravely seriously...so one never needs to reinstall. The only things that I've seen that have ever required action are packages that needed manual configuration updates in /etc (equally solvable by apt purge), and obsolete/broken configuration in /home/user (not solved if this feature request is implemented). What problem is this feature request intended to solve? FrankenDebian? https://wiki.debian.org/DontBreakDebian Cheers, Nicholas P.S. apt install installation-birthday :-)
Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder
I am addressing another case, the one you have not separated partitions for /, /home and swap. As a matter of fact if the installer is able to recognize the home folder having /home separated in another partition is not necessary anymore. The advantage respect having the /home separated, specifically for a desktop use are noteworthy. If the installer, instead of creating /, /home and swap, creates just / and a swap file and if is able to reinstall itself without overwriting the home folder I think is a huge improvement. As a matter of fact if you reinstall Debian, even with /home in another partition, there is not any assisted aid that explain you how to properly setup the /home partition. Having the system partitioned is already a setup for advanced user. Thanks, D. On 9/27/19 6:45 PM, Geert Stappers wrote: On Fri, Sep 27, 2019 at 05:19:06PM -0400, Daniel wrote: Holger Wansing wrote: The debian-installer supports similar use case via the "separate partition for /home" approach. to reinstall Debian on top of itself without overwriting the home partition. Yes, that is what Holger is telling. Groeten Geert Stappers
Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder
Geert Stappers writes: > On Fri, Sep 27, 2019 at 05:19:06PM -0400, Daniel wrote: >> Holger Wansing wrote: >> > The debian-installer supports similar use case via the "separate >> > partition for /home" approach. >> to reinstall Debian on top of itself without overwriting the home partition. > > Yes, that is what Holger is telling. I think Daniel is requesting an option that does something like this: find /install-target -maxdepth 1 | grep -v 'home\|lost+found' | xargs rm -rf Maybe this way isn't robust enough, but active mounts shouldn't have their mount points removed, because rm: cannot remove '/install-target/foo': Device or resource busy BTW, Daniel, you can decruft your system with "apt purge --autoremove foo", which also deletes config in /etc and will notify you if any files remain in /var. One of the greatest strengths of Debian is that unlike other operating systems, smooth upgrades between stable versions are taken seriously...gravely seriously...so one never needs to reinstall. The only things that I've seen that have ever required action are packages that needed manual configuration updates in /etc (equally solvable by apt purge), and obsolete/broken configuration in /home/user (not solved if this feature request is implemented). What problem is this feature request intended to solve? FrankenDebian? https://wiki.debian.org/DontBreakDebian Cheers, Nicholas P.S. apt install installation-birthday :-) signature.asc Description: PGP signature
Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder
On Fri, Sep 27, 2019 at 05:19:06PM -0400, Daniel wrote: > Holger Wansing wrote: > > The debian-installer supports similar use case via the "separate > > partition for /home" approach. > to reinstall Debian on top of itself without overwriting the home partition. Yes, that is what Holger is telling. Groeten Geert Stappers -- Leven en laten leven
Bug#935931: Re: Bug#935931: debian-installer: Reinstalling Debian on a current Debian installation without erasing or fomatting the home folder
Dear All, In this case I suggest hence to expand the use cases and to evaluate the feature to reinstall Debian on top of itself without overwriting the home partition. While it doesn't seem a complicated achievement, it brings instead a great advantage for the end user and improves greatly the desktop experience, for new users as well for the power users. My regards, Daniel
