Re: quotas and reservations (was Re: Looking at RAID)

[Craig Sanders via luv-main]

On Sun, Mar 11, 2018 at 07:56:37PM +1100, Russell Coker wrote:
> On Sunday, 11 March 2018 6:37:56 PM AEDT Craig Sanders via luv-main wrote:
> > On Fri, Mar 09, 2018 at 07:49:48PM +1100, Russell Coker wrote:
> > > Unlike BTRFS, you can expect every feature of ZFS to just work.  It may be
> > > a total PITA to get it working, it may not be something you even want to
> >
> > ZFS just works, and I've never found it to be a PITA at all.  I've been
> > using it since around 2010.
>
> BTRFS snapshots just appear ready to use.  ZFS snapshots have to have extra
> stuff done to expose them.  Snapshots of ZFS block devices have similar levels
> of pain.

What, you mean `zfs set snapdir=visible pool/dataset` or `zfs set
snapdev=visible pool/zvol` is difficult?

(like most other attributes, these are inherited by default by child datasets,
and can be enabled/disabled individually)

The only thing difficult about visible snapshots in ZFS is realising that
tools like rsync will recurse into the .zfs/ snapshot directory unless you
tell them not to. e.g. for rsync you can use a .rsync-filter file like this:

$ cat /.rsync-filter
-r .zfs/***
-s .zfs/***
- /tmp/***

See http://blog.taz.net.au/2012/04/01/rsync-and-zfs-snapshot-directories/


> >  * non-rootfs storage just works - no hassle, no fuss, no drama. Couldn't
> > be easier...certainly a lot less work/hassle than LVM and/or mdam + ext4
> > or xfs.
>
> If the installer supported it then that would be the case.  As an aside does
> the Ubuntu installer support ZFS?

no idea. the last time I used an ubuntu installer was long before they started
supporting ZFS.

> But given that you can just install straight to mdadm+LVM+Ext4 or BTRFS, in
> Debian at least ZFS is much harder to use.

We obviously have different definitions of 'difficult to use'.

I'd agree with "a bit more difficult to set up". but definitely not with "much
harder to use".

There are enough howtos and blog posts and answers on sites like stackexchange
detailing how to get your rootfs on ZFS that it doesn't even qualify as "much
harder to setup", only "a bit harder".

When it comes to actually using either ZFS or LVM to administer your systems
and organise your filesystems/datasets & partitions/zvols, ZFS wins for ease
of use - it's not even a contest.

LVM was good in it's day, but that day was over long ago.


> >  * getting linux to boot from ZFS requires some work (unless your distro
> > supports it natively, like Ubuntu).  A few hours of reading and about the
> > same of careful, methodical work the first time you do it.
>
> Or you decide that something that requires a few hours of reading might be
> too fragile for your root filesystem.

"requires research and work to set up" does not equal "fragile".

Personally, I think anyone who doesn't take the time to research how to best
use the tools at their disposal, and which of the many tools to use, is
negligent.

ZFS doesn't require research because it's extremely difficult to use or
understand.  It requires research because it can do a lot, and there are many
different ways to configure and use it.  To make good choices you need some
understanding of the concepts and you need some familiarity with the tools.
Any complex/flexible tool will require a similar level of research effort.


> >IMO it's worth the effort just for snapshots & zfs send backups of the
> >rootfs.  All the other ZFS features are just gravy.
>
> You can have a cron job that rsyncs the root on Ext4 to ZFS and then uses
> ZFS snapshots etc for managing backups.

Yes, i still do that on my systems that don't have zfs for the root fs.
rsync's a great tool, but it doesn't come close to `zfs snapshot ...` followed
by `zfs send`.

`zfs send` completes in a tiny fraction of the time that an rsync takes,
with much lower disk I/O.  Incremental daily backups that used to take tens
of minutes or even hours to complete - even with hardly any changes - now
complete in a just few minutes.  I barely even notice when the backup jobs
start at ~1am - they used to be a sign that I should go to bed because my
system will be running at a crawl, thrashing the disks for the next few hours.

This is entirely due to rsync having to examine at least the metadata for each
file in the directory tree being backed up (and possibly compare the block
checksums on both sides of the rsync).  With `zfs send`, ZFS knows **exactly**
which blocks have changed in that snapshot, no lookup or scan required - just
send the changed blocks.

Even the startup time of rsync, while it's doing its initial scan of the
backup path is several times longer than an entire zfs send backup.

As I said, this alone makes it worth using ZFS for the rootfs.

Like LVM, rsync was a great tool for its day.  Unlike LVM, it's still worth
using for some things.  IMO the only reason to use LVM these days is because
you have legacy systems that can't be taken down long enough to convert to
ZFS (but you should be planning to switch them to ZFS 

Re: quotas and reservations (was Re: Looking at RAID)

[Russell Coker via luv-main]

On Sunday, 11 March 2018 6:37:56 PM AEDT Craig Sanders via luv-main wrote:
> On Fri, Mar 09, 2018 at 07:49:48PM +1100, Russell Coker wrote:
> > Unlike BTRFS, you can expect every feature of ZFS to just work.  It may be
> > a total PITA to get it working, it may not be something you even want to
> 
> ZFS just works, and I've never found it to be a PITA at all.  I've been
> using it since around 2010.

BTRFS snapshots just appear ready to use.  ZFS snapshots have to have extra 
stuff done to expose them.  Snapshots of ZFS block devices have similar levels 
of pain.

>  * `apt-get install zfs-dkms zfsutils-linux` (and optionally zfs-zed and
>zfsnap) is no big deal.  It's no more trouble than installing anything
>else.

Well the extra compilation step when updating the kernel and the requirement 
to have GCC installed on systems that otherwise don't need it is annoying.  
But an Ubuntu kernel solves that.

>  * non-rootfs storage just works - no hassle, no fuss, no drama. Couldn't be
> easier...certainly a lot less work/hassle than LVM and/or mdam + ext4 or
> xfs.

If the installer supported it then that would be the case.  As an aside does 
the Ubuntu installer support ZFS?

But given that you can just install straight to mdadm+LVM+Ext4 or BTRFS, in 
Debian at least ZFS is much harder to use.

>  * getting linux to boot from ZFS requires some work (unless your distro
>supports it natively, like Ubuntu).  A few hours of reading and about the
> same of careful, methodical work the first time you do it.

Or you decide that something that requires a few hours of reading might be too 
fragile for your root filesystem.

>IMO it's worth the effort just for snapshots & zfs send backups of the
>rootfs.  All the other ZFS features are just gravy.

You can have a cron job that rsyncs the root on Ext4 to ZFS and then uses ZFS 
snapshots etc for managing backups.

> I don't really care about samba, but one of these days I should look into
> doing NFS exports just by setting an attribute on a dataset in zfs.  For
> now, editing /etc/exports is easy enough.

Good luck with that, it didn't work well for me.

I probably could have got it working without too much effort, but the 
alternative of just editing /etc/exports and running "exportfs -r" is very 
easy so I gave up.

-- 
My Main Blog http://etbe.coker.com.au/
My Documents Bloghttp://doc.coker.com.au/

___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

Re: quotas and reservations (was Re: Looking at RAID)

[Craig Sanders via luv-main]

On Fri, Mar 09, 2018 at 07:49:48PM +1100, Russell Coker wrote:

> Unlike BTRFS, you can expect every feature of ZFS to just work.  It may be
> a total PITA to get it working, it may not be something you even want to

ZFS just works, and I've never found it to be a PITA at all.  I've been using
it since around 2010.

 * `apt-get install zfs-dkms zfsutils-linux` (and optionally zfs-zed and
   zfsnap) is no big deal.  It's no more trouble than installing anything
   else.

 * non-rootfs storage just works - no hassle, no fuss, no drama. Couldn't be
   easier...certainly a lot less work/hassle than LVM and/or mdam + ext4 or
   xfs.

 * getting linux to boot from ZFS requires some work (unless your distro
   supports it natively, like Ubuntu).  A few hours of reading and about the
   same of careful, methodical work the first time you do it.

   IMO it's worth the effort just for snapshots & zfs send backups of the
   rootfs.  All the other ZFS features are just gravy.

> use (EG ZFS control of Samba configuration), but it will work if you do the
> right thing.

I've never used that.  Or the auto-NFS stuff. I Never saw any need to try it
since I was already running NFS and Samba - continuing to use what I already
had was no trouble at all :)

I don't really care about samba, but one of these days I should look into
doing NFS exports just by setting an attribute on a dataset in zfs.  For now,
editing /etc/exports is easy enough.

craig

--
craig sanders 
___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

Re: quotas and reservations (was Re: Looking at RAID)

[Russell Coker via luv-main]

On Friday, 9 March 2018 1:04:13 PM AEDT Craig Sanders via luv-main wrote:
> I forgot to mention one very useful difference between 1. partitions & LVM
> Logical Volumes (LV) and 2. btrfs sub-volumes & ZFS datasets.
> 
> Partitions & LVs are created with a fixed size.  The size can be changed
> later, but if you create a 50G /data partittion and a 100G /home partition
> it's easily possible to have one or the other partition being full while the
> other has plenty of space.
> 
> Changing the size of a partition/LV requires changing both the size of the
> partition or LV AND changing the size of the filesystem that's on
> it.  This isn't too difficult but does require some caution and attention to
> detail.  It's recommended to make sure you have a recent backup in case you
> make a mistake.

Some filesystems like XFS don't support reducing the size at all.  So LVM in 
that case is good for extending but can never reduce the size.

Older filesystems like XFS and ext[234] have a fixed table of Inodes which 
will be increased linearly with increases in overall size which may or may not 
be what you want.

> btrfs has btrfs-quota and btrfs-qgroup commands to control subvolume size
> limits.  I've never used them, so can't say any more than that they exist.
> 
> BTW, The btrfs-quota man page says "When the quotas are turned on, they
> affect all extent processing, taking a performance hit. It is not
> recommended to turn on qgroups unless the user intends to actually use
> them."

They probably don't work well due to lack of testing.  Last time I played with 
them I found a bug that made them unusable on my system in 5 minutes.

> With ZFS, you can set a maximum size (quota) for a dataset (e.g. you might
> limit đata/videos to using no more than 200G). You can also reserve space in
> the pool for a dataset to prevent any other dataset from using the reserved
> space - i.e. to guarantee that a dataset will have a minimum amount of
> space available to it, no matter what else is stored on the pool.

Unlike BTRFS, you can expect every feature of ZFS to just work.  It may be a 
total PITA to get it working, it may not be something you even want to use (EG 
ZFS control of Samba configuration), but it will work if you do the right 
thing.

-- 
My Main Blog http://etbe.coker.com.au/
My Documents Bloghttp://doc.coker.com.au/

___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

quotas and reservations (was Re: Looking at RAID)

[Craig Sanders via luv-main]

I forgot to mention one very useful difference between 1. partitions & LVM
Logical Volumes (LV) and 2. btrfs sub-volumes & ZFS datasets.

Partitions & LVs are created with a fixed size.  The size can be changed
later, but if you create a 50G /data partittion and a 100G /home partition
it's easily possible to have one or the other partition being full while the
other has plenty of space.

Changing the size of a partition/LV requires changing both the size of the
partition or LV AND changing the size of the filesystem that's on
it.  This isn't too difficult but does require some caution and attention to
detail.  It's recommended to make sure you have a recent backup in case you
make a mistake.

(LVs can also be thin-provisioned - kind of like a sparse file.  This allows
over-committing of space, e.g. for VM images where you want to allocate a 5 or
10GB virtual disk without wasting space if some VMs only use a GB or two)



With sub-volumes and datastets, by default they don't have any particular
limit - all sub-volumes/datasets share the pool equally.

btrfs has btrfs-quota and btrfs-qgroup commands to control subvolume size
limits.  I've never used them, so can't say any more than that they exist.

BTW, The btrfs-quota man page says "When the quotas are turned on, they affect
all extent processing, taking a performance hit. It is not recommended to turn
on qgroups unless the user intends to actually use them."


With ZFS, you can set a maximum size (quota) for a dataset (e.g. you might
limit đata/videos to using no more than 200G). You can also reserve space in
the pool for a dataset to prevent any other dataset from using the reserved
space - i.e. to guarantee that a dataset will have a minimum amount of space
available to it, no matter what else is stored on the pool.

You can also change the size limit and/or the reservation at any time, e.g. to
change the quotæ for data/videos from 200GB to 250GB is one simple command:
zfs set quota=250G data/videos

I've rarely used reservations, but frequently set size limits on datasets.


In short, partitions have "hard" or "fixed" limits, subvolumes and datasets have
"soft" limits.

This seems pretty simple but eliminates the most common reason for
wanting/needing to mess around with your filesystems after you've created
them.  Running out of space on / or /home or /var while there's still plenty
of space in other partitions is quite common.

craig

ps: ZFS also supports traditional unix user & group quotas. btrfs doesn't.
This isn't much use for a home system, but can be useful for organisations to
prevent indvidual users from hogging all the disk space.

--
craig sanders 
___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main