Re: [Discuss] Backing up the entire filesystem
Hi, Shirley Márquez Dúlcey writes: > Another thing to keep in mind is that ZFS does have one flaw; it's a > memory hog. If you have a large ZFS filesystem you will need a LOT of > RAM to get acceptable performance. But it does represent the current > state of the art for file system data integrity. I think current standard is 1GB RAM per TB of disk, or 5GB/TB if you have dedup turned on. > I have to allow that my only experience with ZFS to date is with > FreeNAS, which is based on FreeBSD. I have moved all my bulk data > storage to a pair of NAS boxes and have a relatively small amount of > local space on each computer. FreeNAS does not use ZFS for the system > volume. Eh? My FreeNAS system uses ZFS on the boot disk? -derek -- Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory Member, MIT Student Information Processing Board (SIPB) URL: http://web.mit.edu/warlord/PP-ASEL-IA N1NWH warl...@mit.eduPGP key available ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
On Fri, Oct 26, 2018 at 02:30:10PM -0400, Marco Milano wrote: > > > On 10/26/18 2:22 PM, Dan Ritter wrote: > > > > > > > > On 10/26/18 1:55 PM, Shirley Márquez Dúlcey wrote: > > > > > Another thing to keep in mind is that ZFS does have one flaw; it's a > > > > > memory hog. If you have a large ZFS filesystem you will need a LOT of > > > > > RAM to get acceptable performance. But it does represent the current > > > > > state of the art for file system data integrity. > > > > > > > > I think as long as you don't use dedup, it works perfectly fine > > > > on a system with 8GB or 16GB RAM. > > > > The rule of thumb is 1GB per TB of used space, so for Shirley's > > NAS boxes, dedup would actually work. I don't recommend it, > > though. > > > > I am configuring a system with 200TB zfs data storage, with only 32GB of RAM, > ZFS with no dedup, lz4 compression, ubuntu 18.04 server, I am pretty confident > that it will work perfectly fine. Yes, the rule of thumb above is the RAM required for dedup, not anything else. -dsr- ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
On 10/26/18 2:22 PM, Dan Ritter wrote: On 10/26/18 1:55 PM, Shirley Márquez Dúlcey wrote: Another thing to keep in mind is that ZFS does have one flaw; it's a memory hog. If you have a large ZFS filesystem you will need a LOT of RAM to get acceptable performance. But it does represent the current state of the art for file system data integrity. I think as long as you don't use dedup, it works perfectly fine on a system with 8GB or 16GB RAM. The rule of thumb is 1GB per TB of used space, so for Shirley's NAS boxes, dedup would actually work. I don't recommend it, though. I am configuring a system with 200TB zfs data storage, with only 32GB of RAM, ZFS with no dedup, lz4 compression, ubuntu 18.04 server, I am pretty confident that it will work perfectly fine. -- Marco ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
On Fri, Oct 26, 2018 at 02:18:12PM -0400, Shirley Márquez Dúlcey wrote: > My NAS boxes each have 16GB. One has two 4TB and two 3TB drives in > mirrored pairs; the other has three 1.5TB drives in a RAIDZ1 setup. > And that's just for being a NAS. I would probably need more RAM, > especially on the one with four drives, if I were also running > applications on them. > > Why two smaller boxes rather than one big one? Mostly because I > already own a pair of low end AMD systems using mini-ITX motherboards > and AM1 socket CPUs that I originally got for another purpose but was > no longer using, and I already had all the drives. Those motherboards > only have two DDR3 memory sockets and thus have a RAM ceiling of 16GB, > and the cases will only hold four hard drives. > On Fri, Oct 26, 2018 at 2:04 PM Marco Milano wrote: > > > > > > > > On 10/26/18 1:55 PM, Shirley Márquez Dúlcey wrote: > > > Another thing to keep in mind is that ZFS does have one flaw; it's a > > > memory hog. If you have a large ZFS filesystem you will need a LOT of > > > RAM to get acceptable performance. But it does represent the current > > > state of the art for file system data integrity. > > > > I think as long as you don't use dedup, it works perfectly fine > > on a system with 8GB or 16GB RAM. The rule of thumb is 1GB per TB of used space, so for Shirley's NAS boxes, dedup would actually work. I don't recommend it, though. -dsr- ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
On Fri, Oct 26, 2018 at 01:55:57PM -0400, Shirley Márquez Dúlcey wrote: > Another thing to keep in mind is that ZFS does have one flaw; it's a > memory hog. If you have a large ZFS filesystem you will need a LOT of > RAM to get acceptable performance. But it does represent the current > state of the art for file system data integrity. It only looks like a memory hog. ZFS correctly marks all of it's ARC as cache, so it's mostly available when something else asks for it. If you feel concerned, you can limit usage with with options zfs zfs_arc_max=8589934592 (or however many bytes you want to limit it to) -dsr- ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
My NAS boxes each have 16GB. One has two 4TB and two 3TB drives in mirrored pairs; the other has three 1.5TB drives in a RAIDZ1 setup. And that's just for being a NAS. I would probably need more RAM, especially on the one with four drives, if I were also running applications on them. Why two smaller boxes rather than one big one? Mostly because I already own a pair of low end AMD systems using mini-ITX motherboards and AM1 socket CPUs that I originally got for another purpose but was no longer using, and I already had all the drives. Those motherboards only have two DDR3 memory sockets and thus have a RAM ceiling of 16GB, and the cases will only hold four hard drives. On Fri, Oct 26, 2018 at 2:04 PM Marco Milano wrote: > > > > On 10/26/18 1:55 PM, Shirley Márquez Dúlcey wrote: > > Another thing to keep in mind is that ZFS does have one flaw; it's a > > memory hog. If you have a large ZFS filesystem you will need a LOT of > > RAM to get acceptable performance. But it does represent the current > > state of the art for file system data integrity. > > I think as long as you don't use dedup, it works perfectly fine > on a system with 8GB or 16GB RAM. > > -- Marco > > ___ > Discuss mailing list > Discuss@blu.org > http://lists.blu.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
On 10/26/18 1:55 PM, Shirley Márquez Dúlcey wrote: Another thing to keep in mind is that ZFS does have one flaw; it's a memory hog. If you have a large ZFS filesystem you will need a LOT of RAM to get acceptable performance. But it does represent the current state of the art for file system data integrity. I think as long as you don't use dedup, it works perfectly fine on a system with 8GB or 16GB RAM. -- Marco ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
Another thing to keep in mind is that ZFS does have one flaw; it's a memory hog. If you have a large ZFS filesystem you will need a LOT of RAM to get acceptable performance. But it does represent the current state of the art for file system data integrity. I have to allow that my only experience with ZFS to date is with FreeNAS, which is based on FreeBSD. I have moved all my bulk data storage to a pair of NAS boxes and have a relatively small amount of local space on each computer. FreeNAS does not use ZFS for the system volume. On Fri, Oct 26, 2018 at 1:40 PM Shirley Márquez Dúlcey wrote: > > I don't think any of the usual bootloaders support ZFS. The usual way > to handle that is to create a small /boot filesystem using something > else like ext4, just like you did back in the days when large hard > drives weren't supported by the BIOS and therefore the bootloader > couldn't access their entire capacity. The installers for most distros > can handle setting up a system that way. > On Fri, Oct 26, 2018 at 12:32 PM Marco Milano wrote: > > > > > > > > On 10/26/18 11:38 AM, Rich Pieri wrote: > > > > > > > > Having recently-ish rebuilt my home server with full ZFS I have to say > > > that ZFS makes backups ridiculously easy. And quick. > > > > How did you get the root filesystem under ZFS? > > > > Is there a nice/clean/easy recipe ? > > > > -- Marco > > > > > > ___ > > Discuss mailing list > > Discuss@blu.org > > http://lists.blu.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
I don't think any of the usual bootloaders support ZFS. The usual way to handle that is to create a small /boot filesystem using something else like ext4, just like you did back in the days when large hard drives weren't supported by the BIOS and therefore the bootloader couldn't access their entire capacity. The installers for most distros can handle setting up a system that way. On Fri, Oct 26, 2018 at 12:32 PM Marco Milano wrote: > > > > On 10/26/18 11:38 AM, Rich Pieri wrote: > > > > > Having recently-ish rebuilt my home server with full ZFS I have to say > > that ZFS makes backups ridiculously easy. And quick. > > How did you get the root filesystem under ZFS? > > Is there a nice/clean/easy recipe ? > > -- Marco > > > ___ > Discuss mailing list > Discuss@blu.org > http://lists.blu.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
On Fri, 26 Oct 2018 12:32:26 -0400 Marco Milano wrote: > How did you get the root filesystem under ZFS? > > Is there a nice/clean/easy recipe ? Debian does not include ZFS so not "clean". You have to do a normal install (on XFS), install the ZFS on Linux packages, build your zpool, copy XFS to ZFS, update the initrd to include SPL and ZFS kernel modules, set the zpool's mount point to /, and test. It's easier with Ubuntu since ZFSoL is included but still not clean because Ubuntu's installers don't know about ZFS so you still have to hack it by hand. To be honest I don't think it's worth the effort. ext2 for /boot (if you have it) and XFS for / is fine for the OS since these in principle don't need routine backups. Run etckeeper to put /etc under automatic revision control, and rsync to durable storage as needed. -- Rich Pieri ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
On 10/26/18 11:38 AM, Rich Pieri wrote: Having recently-ish rebuilt my home server with full ZFS I have to say that ZFS makes backups ridiculously easy. And quick. How did you get the root filesystem under ZFS? Is there a nice/clean/easy recipe ? -- Marco ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
On Fri, 26 Oct 2018 05:15:51 -0700 Rich Braun wrote: > The ext4 filesystem has been the mainstream default on most Linux > distros since 2008. I’ve been using a large number of systems using RHEL 7's default is XFS. SLES 12's and SLES 15's default is a mix of Btrfs and XFS. > this filesystem, and a lesser number with xfs, ever since. Never once > have I had filesystem corruption like with previous filesystems; > there are other far-more likely data loss scenarios (like, uh, human > error). I've been unfortunate, sure enough, but the instances of extent corruption I've had to clean up were not due to user error. Regardless of the causes, the tools available are incapable of salvaging corrupt ext4 filesystems in that state. And since the tools don't do what they need to do I will maintain that you use ext4 at your own risk. > I have published a python package ‘secondshot’ to add a few > capabilities on top of rsnapshot; see my repos on GitHub / > dockerhub / pypi. (github.com/instantlinux). Having recently-ish rebuilt my home server with full ZFS I have to say that ZFS makes backups ridiculously easy. And quick. -- Rich Pieri ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
Re: [Discuss] Backing up the entire filesystem
Rich Pieri notes: > there is a flaw (or possibly several) in the ext4 [that] causes extent > corruption and data loss. > fsck will *not* repair an ext4 filesystem in this state. The ext4 filesystem has been the mainstream default on most Linux distros since 2008. I’ve been using a large number of systems using this filesystem, and a lesser number with xfs, ever since. Never once have I had filesystem corruption like with previous filesystems; there are other far-more likely data loss scenarios (like, uh, human error). I could not find a reference for the “fatal flaw” cited here but here’s an overview of how ext4 stacks up: https://opensource.com/article/18/4/ext4-filesystem. I’d say if you’re running a mainstream distro with ext4, do your backups, don’t worry about corruption (unless you’re inside the Beltway), and leave the religious flame wars about with filesystem is best to the Linux kernel-contributor experts. If you’re setting up a new system, xfs, btrfs and ext4 are all valid choices. On the main question of backups: just a couple days ago, Crashplan finally pulled the plug on its Home service. I have two other solutions active for my systems: Duplicati for backup to a local server and to the Backblaze B2 cloud service ($60/year per terabyte), and rsnapshot to another local server. I have published a python package ‘secondshot’ to add a few capabilities on top of rsnapshot; see my repos on GitHub / dockerhub / pypi. (github.com/instantlinux). -rich ___ Discuss mailing list Discuss@blu.org http://lists.blu.org/mailman/listinfo/discuss
