Thanks for picking this up! Looking forward to not searching the web/our forum for the good answers to questions that come up quite often.
a few mostly stylistic (as in more a matter of my taste) comments inline: On Fri, 17 Jul 2020 14:12:32 +0200 Aaron Lauterer <[email protected]> wrote: > This new section explains the performance and failure properties of > mirror and RAIDZ VDEVs as well as the "unexpected" higher space usage by > ZVOLs on a RAIDZ. > > Signed-off-by: Aaron Lauterer <[email protected]> > --- > > This is a first draft to explain the performance characteristics of the > different RAID levels / VDEV types, as well as their failure behavior. > > Additionally it explains the situation why a VM disk (ZVOL) can end up > using quite a bit more space than expected when placed on a pool made of > RAIDZ VDEVs. > > The motivation behind this is, that in the recent past, these things > came up quite a bit. Thus it would be nice to have some documentation > that we can link to and having it in the docs might help users to make > an informed decision from the start. > > I hope I did not mess up any technical details and that it is > understandable enough. > > local-zfs.adoc | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 96 insertions(+) > > diff --git a/local-zfs.adoc b/local-zfs.adoc > index fd03e89..48f6540 100644 > --- a/local-zfs.adoc > +++ b/local-zfs.adoc > @@ -151,6 +151,102 @@ rpool/swap 4.25G 7.69T 64K - > ---- > > > +[[sysadmin_zfs_raid_considerations]] > +ZFS RAID Level Considerations > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +There are a few factors to take into consideration when choosing the layout > of > +a ZFS pool. > + > + > +[[sysadmin_zfs_raid_performance]] > +Performance > +^^^^^^^^^^^ > + > +Different types of VDEVs have different performance behaviors. The two we have a few mentions of vdev (written without caps in the system-booting.adoc) - for consistency either write it small here as well or change the system-booting part. as for the content - a short explanation what a vdev is might be helpful, and mentioning that all top level vdevs in a pool are striped (as in RAID0) together > +parameters of interest are the IOPS (Input/Output Operations per Second) and > the > +bandwidth with which data can be written or read. > + > +A 'mirror' VDEV will approximately behave like a single disk in regards to > both > +parameters when writing data. When reading data if will behave like the > number > +of disks in the mirror. in the section above (and in the installer and disk management GUI we talk about RAIDX - maybe refer to this at least in paranthesis: A 'mirror' VDEV (RAID1) .... > + > +A common situation is to have 4 disks. When setting it up as 2 mirror VDEVs > the here the same with RAID10 > +pool will have the write characteristics as two single disks in regard of > IOPS > +and bandwidth. For read operations it will resemble 4 single disks. > + > +A 'RAIDZ' of any redundancy level will approximately behave like a single > disk > +in regard of IOPS with a lot of bandwidth. How much bandwidth depends on the > +size of the RAIDZ VDEV and the redundancy level. > + > +For running VMs, IOPS is the more important metric in most situations. > + > + > +[[sysadmin_zfs_raid_size_space_usage_redundancy]] > +Size, Space usage and Redundancy > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > + > +While a pool made of 'mirror' VDEVs will have the best performance > +characteristics, the usable space will be 50% of the disks available. Less > if a > +mirror VDEV consists of more than 2 disks. To stay functional it needs to > have maybe add (e.g. a 3-way mirror) after 2 disks s/To stay functional it needs to have at least one disk per mirror VDEV available/At least one healthy disk per mirror is needed for the pool to work/ ? > +at least one disk per mirror VDEV available. The pool will fail once all > disks > +in a mirror VDEV fail. maybe drop the last sentence > + > +When using a pool made of 'RAIDZ' VDEVs the usable space to disk ratio will > be > +better in most situations than using mirror VDEVs. This is especially true > when Why not actively describe the usable space: The usable space of a 'RAIDZ' type VDEV of N disks is roughly N-X, with X being the RAIDZ-level. The RAIDZ-level also indicates how many arbitrary disks can fail without losing data. (and drop the redundancy sentence below > +using a large number of disks. A special case is a 4 disk pool with RAIDZ2. > In > +this situation it is usually better to use 2 mirror VDEVs for the better > +performance as the usable space will be the same. In a RAIDZ VDEV, any drive > +can fail and it will stay operational. The number of sustainable drive > failures > +is defined by the redundancy level, a RAIDZ1 can survive the loss of 1 disk, > +consequently, a RAIDZ2 the loss of 2 and a RAIDZ3 the loss of 3 disks. > + > +Another important factor when using any RAIDZ level is how ZVOL datasets, > which > +are used for VM disks, behave. For each data block the pool needs parity data > +which are at least the size of the minimum block size defined by the `ashift` which _is_ at least the size of? > +value of the pool. With an ashift of 12 the block size of the pool is 4k. > The > +default block size for a ZVOL is 8k. Therefore, in a RAIDZ2 each 8k block > +written will cause two additional 4k parity blocks to be written, > +8k + 4k + 4k = 16k. This is of course a simplified approach and the real > +situation will be slightly different with metadata, compression and such not > +being accounted for in this example. > + > +This behavior can be observed when checking the following properties of the > +ZVOL: > + > + * `volsize` > + * `refreservation` (if the pool is not thin provisioned) > + * `used` (if the pool is thin provisioned and without snapshots present) > + > +---- > +# zfs get volsize,refreservation,used /<pool>/vm-<vmid>-disk-X > +---- the '/' in the beginning should be dropped > + > +`volsize` is the size of the disk as it is presented to the VM, while > +`refreservation` shows the reserved space on the pool which includes the > +expected space needed for the parity data. If the pool is thin provisioned, > the > +`refreservation` will be set to 0. Another way to observe the behavior is to > +compare the used disk space within the VM and the `used` property. Be aware > +that snapshots will skew the value. > + > +To counter this effect there are a few options. s/this effect/the increased use of space/ > + > +* Increase the `volblocksize` to improve the data to parity ratio > +* Use 'mirror' VDEVs instead of 'RAIDZ' > +* Use `ashift=9` (block size of 512 bytes) > + > +The `volblocksize` property can only be set when creating a ZVOL. The default > +value can be changed in the storage configuration. When doing this, the guest > +needs to be tuned accordingly and depending on the use case, the problem of > +write amplification if just moved from the ZFS layer up to the guest. > + > +A pool made of 'mirror' VDEVs has a different usable space and failure > behavior > +than a 'RAIDZ' pool. This is already explained above? Maybe add a short recommendation - 'RAID10 has favorable behavior for VM workloads - use RAID10, unless your environment has specific needs and characteristics where RAIDZ performance characteristics are acceptable' ? > + > +Using `ashift=9` when creating the pool can lead to bad > +performance, depending on the disks underneath, and cannot be changed later > on. > + > + > Bootloader > ~~~~~~~~~~ > _______________________________________________ pve-devel mailing list [email protected] https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
