Re: [gentoo-user] Re: Suggestions for backup scheme?
On 06/02/2024 16:19, J. Roeleveld wrote: Ah! Got it. That's one of the things I've been trying to figure out this entire thread, do I need to switch home and root to ZFS to take advantage of its snapshot support for backups? In the case you're describing the "source" filesystem(s) can be anything. It's only the _backup_ filesystem that needs to be ZFS (or similar). If you want to use snapshots, the filesystem will need to support it. (either LVM or ZFS). If you only want to create snapshots on the backupserver, I actually don't see much benefit over using rsync. Because snapshotting uses so much less space? So much so that, for normal usage, I probably have no need to delete any snapshots, for YEARS? Okay, space is not an expensive commodity, and you don't want too many snapshots, simply because digging through all those snapshots would be a nightmare, but personally I wouldn't use a crude rsync simply because I prefer to be frugal in my use of resources. Cheers, Wol
Re: [gentoo-user] Re: Suggestions for backup scheme?
On 06/02/2024 15:35, Grant Edwards wrote: If (like rsnapshot/rsync's hard-link scheme) ZFS snapshots are normal directory trees that can be "browsed" with normal filesystem tools, that would be ideal. [I'll do some googling...] Bear in mind I'm talking lvm snapshots, not ZFS ... And you can configure snapshots to grow as required. I know it's nothing really to do with backups, but if you read the raid wiki page https://raid.wiki.kernel.org/index.php/System2020 - that's how I set up my system, with a smattering of lvm. Just look at the sections pvcreate, vgcreate, lvcreate, it'll tell you how to create the lvm. Then you just format your lvcreate'd partitions, and you can mount them, snapshot them, whatever them. So you can either have one backup partition per source partition, or one backup partition and copy all your sources into just that one. Your choice :-) Cheers, Wol
Re: [gentoo-user] Re: Suggestions for backup scheme?
On 06/02/2024 13:12, J. Roeleveld wrote: Clearly Oracle likes this state of affairs. Either that, or they are encumbered in some way from just GPLing the ZFS code. Since they on paper own the code for both projects it seems crazy to me that this situation persists. GPL is not necessarily the best license for releasing code. I've got some private projects that I could publish. But before I do that, I'd have to decide on a License. I would prefer something other than GPL. Okay. What do you want to achieve. Let's just lump licences into two categories to start with and ask the question "Who do you want to free?" If that sounds weird, it's because both Copyleft and Permissive claim to be free, but have completely different target audiences. Once you've answered that question, it'll make choosing a licence so much easier. GPL gives freedom to the END USER. It's intended to protect the users of your program from being held to ransom. Permissive gives freedom to the DEVELOPER. It's intended to let other programmers take advantage of your code and use it. Once you've decided what sort of licence you want, it'll be easier to decide what licence you want. Cheers, Wol
[gentoo-user] 147 .pem files in /etc/ need updating
After a routine update this morning (last one was probably 3 days ago), I see that 147 files in /etc need updating. When I run etc-update, they're all ".pem" CA files (or links?). It looks like it was all of the .pem files under /etc/ssl/certs. I did a -5, and all seems well. It's a bit alarming when portage tells you that all of your root CA files have been modified since they were installed. The package app-misc/ca-certificates was updated last week (and it's quite possible the 147 /etc files needed updating after that), but I've never had to deal with certificate file updates like that before (nor have I ever seen anything similar on other Gentoo machines). The only thing I can think of that seems relevent is that last week I did add one _local_ certificate using the method prescribed by the Wiki: # mkdir -p /usr/local/share/ca-certificates # cp .crt /usr/local/share/ca-certificates # update-ca-certificates Would that fool portage into thinking that all 147 CA files belonging to app-misc/ca-certificates had been modified and needed to be "merged" when app-misc/ca-certificates got updgraded? -- Grant
[gentoo-user] Re: Suggestions for backup scheme?
On 2024-02-06, J. Roeleveld wrote: > If you want to use snapshots, the filesystem will need to support it. (either > LVM or ZFS). If you only want to create snapshots on the backupserver, I > actually don't see much benefit over using rsync. Upthread I've been told that ZFS snapshots 1. Require far less disk space than rsync's snapshots. 2. Are far faster. 3. Are atomic. >> If (like rsnapshot/rsync's hard-link scheme) ZFS snapshots are normal >> directory trees that can be "browsed" with normal filesystem tools, >> that would be ideal. [I'll do some googling...] > > ZFS snapshots can be accessed using normal tools and can even be exposed over > NFS mounts making it super easy to find the files again. > > They are normally not visible though, you need to access them specifically > using "/filesystem/path/.zfs/snapshot" Great, that's exactly what I would hope for. I'm reading up on ZFS, and from what I've gleaned so far, it seems lake ZFS source and ZFS backup certainly would be ideal. It's almost like the ZFS filesystem designers had thought about "how to backup" from the start. Something that all of the old-school filesystem designers clearly hadn't. :)
[gentoo-user] Re: Suggestions for backup scheme?
On 2024-02-06, J. Roeleveld wrote: > On Tuesday, February 6, 2024 4:38:11 PM CET Grant Edwards wrote: >> On 2024-02-05, J. Roeleveld wrote: >> > On Wednesday, January 31, 2024 6:56:47 PM CET Rich Freeman wrote: >> >> On Wed, Jan 31, 2024 at 12:40 PM Thelma wrote: >> >> > If zfs file system is superior to ext4 and it seems to it is. >> >> > Why hasn't it been adopted more widely in Linux? >> >> >> >> The main barrier is that its license isn't GPL-compatible. It is >> >> FOSS, but the license was basically designed to keep it from being >> >> incorporated into the mainline kernel. >> > >> > Which isn't as much of an issue as it sounds. You can still add it >> > into the initramfs and can easily load the module. >> >> What if you don't use an initrd? >> >> I presume that boot/root on ext4 and home on ZFS would not require an >> initrd? > > Yes, that wouldn't require an initrd. But why would you limit this? Because I really, really dislike having to use an initrd. That's probably just an irrational 30 year old prejudice, but over the decades I've found live to be far simpler and more pleasant without initrds. Maybe things have improved over the years, but way back when I did use distros that required initrds, they seem to be a constant, nagging source of headaches. > ZFS works best when given the FULL drive. Where do you put swap? > For my server, I use "bliss-initramfs" to generate the initramfs and > have not had any issues with this since I started using ZFS. > > Especially the ease of generating snapshots also make it really easy > to roll back an update if anything went wrong. If your > root-partition isn't on ZFS, you can't easily roll back. True. However, I've never adopted the practice of backing up my root fs (except for a few specific directories like /etc), and haven't ever really run into situations where I wished I had. It's all stuff that can easily be reinstalled. -- Grant
Re: [gentoo-user] Re: Suggestions for backup scheme?
On Tuesday, February 6, 2024 4:35:34 PM CET Grant Edwards wrote: > On 2024-02-05, Wols Lists wrote: > > On 04/02/2024 15:48, Grant Edwards wrote: > >> OK I see. That's a bit different than what I'm doing. I'm backing up > >> a specific set of directory trees from a couple different > >> filesystems. There are large portions of the "source" filesystems that > >> I have no need to back up. And within those directory trees that do > >> get backed up there are also some excluded subtrees. > > > > But my scheme still works here. The filesystem I'm snapshotting is the > > backup. As such, it only contains the stuff I want backed up, copied > > across using rsync. > > > > There's nothing stopping me running several rsyncs from the live system, > > from several different partitions, to the backup partition. > > Ah! Got it. That's one of the things I've been trying to figure out > this entire thread, do I need to switch home and root to ZFS to take > advantage of its snapshot support for backups? In the case you're > describing the "source" filesystem(s) can be anything. It's only the > _backup_ filesystem that needs to be ZFS (or similar). If you want to use snapshots, the filesystem will need to support it. (either LVM or ZFS). If you only want to create snapshots on the backupserver, I actually don't see much benefit over using rsync. > If (like rsnapshot/rsync's hard-link scheme) ZFS snapshots are normal > directory trees that can be "browsed" with normal filesystem tools, > that would be ideal. [I'll do some googling...] ZFS snapshots can be accessed using normal tools and can even be exposed over NFS mounts making it super easy to find the files again. They are normally not visible though, you need to access them specifically using "/filesystem/path/.zfs/snapshot" -- Joost
Re: [gentoo-user] Re: Suggestions for backup scheme?
On Tuesday, February 6, 2024 4:38:11 PM CET Grant Edwards wrote: > On 2024-02-05, J. Roeleveld wrote: > > On Wednesday, January 31, 2024 6:56:47 PM CET Rich Freeman wrote: > >> On Wed, Jan 31, 2024 at 12:40 PM Thelma wrote: > >> > If zfs file system is superior to ext4 and it seems to it is. > >> > Why hasn't it been adopted more widely in Linux? > >> > >> The main barrier is that its license isn't GPL-compatible. It is > >> FOSS, but the license was basically designed to keep it from being > >> incorporated into the mainline kernel. > > > > Which isn't as much of an issue as it sounds. You can still add it > > into the initramfs and can easily load the module. > > What if you don't use an initrd? > > I presume that boot/root on ext4 and home on ZFS would not require an > initrd? Yes, that wouldn't require an initrd. But why would you limit this? ZFS works best when given the FULL drive. For my server, I use "bliss-initramfs" to generate the initramfs and have not had any issues with this since I started using ZFS. Especially the ease of generating snapshots also make it really easy to roll back an update if anything went wrong. If your root-partition isn't on ZFS, you can't easily roll back. -- Joost
[gentoo-user] Re: Suggestions for backup scheme?
On 2024-02-05, J. Roeleveld wrote: > On Wednesday, January 31, 2024 6:56:47 PM CET Rich Freeman wrote: >> On Wed, Jan 31, 2024 at 12:40 PM Thelma wrote: >> > If zfs file system is superior to ext4 and it seems to it is. >> > Why hasn't it been adopted more widely in Linux? >> >> The main barrier is that its license isn't GPL-compatible. It is >> FOSS, but the license was basically designed to keep it from being >> incorporated into the mainline kernel. > > Which isn't as much of an issue as it sounds. You can still add it > into the initramfs and can easily load the module. What if you don't use an initrd? I presume that boot/root on ext4 and home on ZFS would not require an initrd?
[gentoo-user] Re: Suggestions for backup scheme?
On 2024-02-05, Wols Lists wrote: > On 04/02/2024 15:48, Grant Edwards wrote: >> OK I see. That's a bit different than what I'm doing. I'm backing up >> a specific set of directory trees from a couple different >> filesystems. There are large portions of the "source" filesystems that >> I have no need to back up. And within those directory trees that do >> get backed up there are also some excluded subtrees. > > But my scheme still works here. The filesystem I'm snapshotting is the > backup. As such, it only contains the stuff I want backed up, copied > across using rsync. > > There's nothing stopping me running several rsyncs from the live system, > from several different partitions, to the backup partition. Ah! Got it. That's one of the things I've been trying to figure out this entire thread, do I need to switch home and root to ZFS to take advantage of its snapshot support for backups? In the case you're describing the "source" filesystem(s) can be anything. It's only the _backup_ filesystem that needs to be ZFS (or similar). If (like rsnapshot/rsync's hard-link scheme) ZFS snapshots are normal directory trees that can be "browsed" with normal filesystem tools, that would be ideal. [I'll do some googling...] -- Grant
Re: [gentoo-user] Re: Suggestions for backup scheme?
On Monday, February 5, 2024 2:35:12 PM CET Rich Freeman wrote: > First, thanks for the Ars link in the other email. I'll give that a read. You're welcome. I found that when I was looking for the latest state of btrfs. I was actually hoping that the biggest issues had been resolved by now. > On Mon, Feb 5, 2024 at 7:55 AM J. Roeleveld wrote: > > On Wednesday, January 31, 2024 6:56:47 PM CET Rich Freeman wrote: > > > The main barrier is that its license isn't GPL-compatible. It is > > > FOSS, but the license was basically designed to keep it from being > > > incorporated into the mainline kernel. > > > > Which isn't as much of an issue as it sounds. You can still add it into > > the > > initramfs and can easily load the module. > > And the code still works with the functions the kernel devs pushed behind > > the GPL-wall if you simply remove that wall from your own kernel. > > (Which is advisable as it will improve performance) > > So, that's great for random individuals, but companies are going to be > hesitant to do that, especially for anything they redistribute. This > is part of why it isn't mainstream. Not for Linux. *BSD has no such issues and that is why the mainstream SAN/NAS distributions are based on *BSD. (replace '*' with your preferred flavour) > A big part of the reason that Linux is mainstream is that it doesn't > have any legal/license encumbrances. If you have 100 instances of > something and want to have 200 instances, you just turn a dial or add > hardware. There isn't anybody you need to get permission from or pay. > > The result is that the murky legal situation makes ZFS unattractive. > If I were publishing some large commercial software package, I'd > personally be hesitant to embrace ZFS on Linux in it for that reason, > even though I use it all the time personally. Proxmox has ZFS native and afaik, it is using Linux? > > > The odd thing is that right now Oracle controls both ZFS and btrfs, > > > with the latter doing mostly the same thing and being GPL-compatible, > > > but it hasn't tended to be as stable. So we're in a really long > > > transitional period to btrfs becoming as reliable. > > > > After all this time, I have given up on waiting for btrfs. As mentioned in > > my other reply, it's still nowhere near reliable. > > Clearly Oracle likes this state of affairs. Either that, or they are > encumbered in some way from just GPLing the ZFS code. Since they on > paper own the code for both projects it seems crazy to me that this > situation persists. GPL is not necessarily the best license for releasing code. I've got some private projects that I could publish. But before I do that, I'd have to decide on a License. I would prefer something other than GPL. > > To make this easier, there is a compatiblity option when creating a new > > zpool. It's also listed in the zfs-kmod ebuild: > > - zpool create -o compatibility=*grub*2 ... > > - Refer to /usr/share/zfs/compatibility.d/*grub*2 for list of features. > > Oh, that is VERY helpful. I've found random many-years-old webpages > with the appropriate instructions, but something that is part of the > maintained project is much more useful. > > Granted, I think the bottom line is that boot probably shouldn't be on > the same filesystem as large volumes of data, as these feature > restrictions are going to be cumbersome. I'm guessing you can't > shrink vdevs, for example. I actually have the kernel and initramfs on a EFI boot partition and that is enough to get the zpool mounted for use. There is also "ZFSBootMenu" which, afaik, doesn't need this: https://docs.zfsbootmenu.org/en/latest/index.html -- Joost
