Re: More ZFS testing in 9.0_BETA
So ZFS on NetBSD, is this amd64 specific or it works on all other platforms, e.g. sparc64, arm32, arm64, etc? Also, FreeBSD seem to be moving to "ZFS on Linux", anyone knows if NetBSD developers have similar plans in the future?
Re: More ZFS testing in 9.0_BETA
Chavdar Ivanov writes: > On Thu, 15 Aug 2019 at 13:53, Chavdar Ivanov wrote: >> >> FYI - as I mentioned it elsewhere - I modified my qemu-nvmm VMs from >> GPT slices and raw files to zvols without any problems. My next test >> would be to see if I can ISCSI-share a zvol - which should work. > > It does, as expected, but is a bit slow. I tested using something > called IsMyHDOk, the resulting benchmark was 17.7; using a ZVOL > FreeNAS target over the same slowish connection I get 42.2; for > comparison on the same laptop the second rusty SATA HDD gives me > 175.7, whilst the first M.2 SATA gives me 972.1. > > It is in no way the right way to use iSCSI - the laptop is connected > wirelessly to an access point; the adapter clams current speed 866.7 > mb/s (?), which then links to the upstairs room via a powerline > adapter, advertised az 1gb/s, but which has never given me anything > better than about 130mb/s - but then the link to the FreeNAS zvol goes > through the same route and on top of this FreeNAS is a XCP-NG guest, > whereas the NetBSD laptop is using a gigabit Ethernet to a local > switch beyond the powerline adapter... Pretty nice... I have abused the NetBSD iscsi target a couple of times. You can present a USB enclosure via iscsi to whatever can use iscsi and I have done the same with a LVM lv. I would have expected zvols to work in the same way. Glad to hear it does. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: More ZFS testing in 9.0_BETA
On Thu, 15 Aug 2019 at 13:53, Chavdar Ivanov wrote: > > FYI - as I mentioned it elsewhere - I modified my qemu-nvmm VMs from > GPT slices and raw files to zvols without any problems. My next test > would be to see if I can ISCSI-share a zvol - which should work. It does, as expected, but is a bit slow. I tested using something called IsMyHDOk, the resulting benchmark was 17.7; using a ZVOL FreeNAS target over the same slowish connection I get 42.2; for comparison on the same laptop the second rusty SATA HDD gives me 175.7, whilst the first M.2 SATA gives me 972.1. It is in no way the right way to use iSCSI - the laptop is connected wirelessly to an access point; the adapter clams current speed 866.7 mb/s (?), which then links to the upstairs room via a powerline adapter, advertised az 1gb/s, but which has never given me anything better than about 130mb/s - but then the link to the FreeNAS zvol goes through the same route and on top of this FreeNAS is a XCP-NG guest, whereas the NetBSD laptop is using a gigabit Ethernet to a local switch beyond the powerline adapter... > Later. > > On Thu, 15 Aug 2019 at 13:25, Brad Spencer wrote: > > > > > > I did some more ZFS testing in 9.0_BETA, sorry for the length. There > > are a lot of advantages and use cases in getting ZFS more completely > > integrated into the NetBSD universe. Raidframe+LVM is great too, but > > there are things that are harder to do with it. > > > > One of the purposes of these tests was to see how NetBSD ZFS interacted > > with other ZFS implementations in other OSs. The following was done: > > > > "zfs export pool" created from NetBSD and "zfs import pool" into FreeBSD > > and SmartOS. Both of these worked completely as expected. with full > > read and write on the pools. There are ZFS features not enabled on > > pools created with NetBSD, but this does not appear to be a problem. As > > long as a "zfs upgrade pool" is not performed you can import and export > > as much as you wish. I have not tried any Linux implementations. > > > > Going the other direction did not work quite as expected. > > > > On pools created on FreeBSD, the ZFS features that are turned on will > > not allow the pool to import as writable on NetBSD. This is not really > > an unexpected thing, given that features on turned on in FreeBSD that > > are not in NetBSD. The pool will import readonly and appears to work as > > expected in that case. It should be possible to create pools in FreeBSD > > with features disabled that will allow a better import. > > > > For SmartOS, pools created there will (might?? I only use the > > filesystems I presented to the ISO) have a GPT header attached to them. > > When NetBSD sees that, by default, it will create wedges. As a general > > thing, this wouldn't be a problem. However, "zpool import" does not > > appear to see dk devices or refuses to deal with them and thus no pools > > were noticed for import. You can sort of force the issue by creating a > > directory with symlinks to the /dev/dkN devices in them and then do a > > "zpool import -d /dev/directory". When this is attempted, the DOMU > > guest I was using panicked: > > > > panic: kernel diagnostic assertion "seg <= BLKIF_MAX_SEGMENTS_PER_REQUEST" > > failed: file "/usr/src/sys/arch/xen/xen/xbd_xenbus.c", line 1032 > > > > I am running an older 8.99.xx with Xen 4.8.x, but as this was from the > > guest, I suspect that a bug is still present. From the BT, it appears > > that it tried to do the import, but failed somewhere in the lower level > > disk routines. I did not try doing this with FreeBSD. > > > > > > Other things of note: > > > > Assuming you want to have separate /var and /usr, trying to use ZFS for > > those is a challenge. One of the biggest issues is that /zbin/zfs and > > /zbin/zpool have linked against shared libraries in /usr/lib. This is > > probably not for the best. Further, mostly because of this, you can't > > reasonably have a /etc/rc.d/zfs start the module and make it usable for > > critical_filesystems in /etc/rc.conf (you can fake it out by putting the > > contents of the shared libraries in /usr/lib and then mount over it, but > > that is a bit ugly). Having /var and /usr use LVM is possible mostly > > because /etc/rc.d/lvm exists and is started before critical_filesystems. > > ZFS should probably work the same way. > > > > Another item is that mountpoint=legacy really does not work. Right now, > > there is neither support in the general mount system call for ZFS nor > > does a /sbin/mount_zfs exist. You can actually get some of this effect > > by a simple shell script in /sbin called mount_zfs that translates the > > request into a "/sbin/zfs set mountpoint=XXX" call followed by a "zfs > > mount XXX". I wrote a quick and dirty one of these, and with a quick > > and dirty /etc/rc.d/zfs, it will allow the effect (mostly) having /var > > and /usr be from a ZFS pool. > > > > An unexpected problem was with /var in this
Re: More ZFS testing in 9.0_BETA
FYI - as I mentioned it elsewhere - I modified my qemu-nvmm VMs from GPT slices and raw files to zvols without any problems. My next test would be to see if I can ISCSI-share a zvol - which should work. Later. On Thu, 15 Aug 2019 at 13:25, Brad Spencer wrote: > > > I did some more ZFS testing in 9.0_BETA, sorry for the length. There > are a lot of advantages and use cases in getting ZFS more completely > integrated into the NetBSD universe. Raidframe+LVM is great too, but > there are things that are harder to do with it. > > One of the purposes of these tests was to see how NetBSD ZFS interacted > with other ZFS implementations in other OSs. The following was done: > > "zfs export pool" created from NetBSD and "zfs import pool" into FreeBSD > and SmartOS. Both of these worked completely as expected. with full > read and write on the pools. There are ZFS features not enabled on > pools created with NetBSD, but this does not appear to be a problem. As > long as a "zfs upgrade pool" is not performed you can import and export > as much as you wish. I have not tried any Linux implementations. > > Going the other direction did not work quite as expected. > > On pools created on FreeBSD, the ZFS features that are turned on will > not allow the pool to import as writable on NetBSD. This is not really > an unexpected thing, given that features on turned on in FreeBSD that > are not in NetBSD. The pool will import readonly and appears to work as > expected in that case. It should be possible to create pools in FreeBSD > with features disabled that will allow a better import. > > For SmartOS, pools created there will (might?? I only use the > filesystems I presented to the ISO) have a GPT header attached to them. > When NetBSD sees that, by default, it will create wedges. As a general > thing, this wouldn't be a problem. However, "zpool import" does not > appear to see dk devices or refuses to deal with them and thus no pools > were noticed for import. You can sort of force the issue by creating a > directory with symlinks to the /dev/dkN devices in them and then do a > "zpool import -d /dev/directory". When this is attempted, the DOMU > guest I was using panicked: > > panic: kernel diagnostic assertion "seg <= BLKIF_MAX_SEGMENTS_PER_REQUEST" > failed: file "/usr/src/sys/arch/xen/xen/xbd_xenbus.c", line 1032 > > I am running an older 8.99.xx with Xen 4.8.x, but as this was from the > guest, I suspect that a bug is still present. From the BT, it appears > that it tried to do the import, but failed somewhere in the lower level > disk routines. I did not try doing this with FreeBSD. > > > Other things of note: > > Assuming you want to have separate /var and /usr, trying to use ZFS for > those is a challenge. One of the biggest issues is that /zbin/zfs and > /zbin/zpool have linked against shared libraries in /usr/lib. This is > probably not for the best. Further, mostly because of this, you can't > reasonably have a /etc/rc.d/zfs start the module and make it usable for > critical_filesystems in /etc/rc.conf (you can fake it out by putting the > contents of the shared libraries in /usr/lib and then mount over it, but > that is a bit ugly). Having /var and /usr use LVM is possible mostly > because /etc/rc.d/lvm exists and is started before critical_filesystems. > ZFS should probably work the same way. > > Another item is that mountpoint=legacy really does not work. Right now, > there is neither support in the general mount system call for ZFS nor > does a /sbin/mount_zfs exist. You can actually get some of this effect > by a simple shell script in /sbin called mount_zfs that translates the > request into a "/sbin/zfs set mountpoint=XXX" call followed by a "zfs > mount XXX". I wrote a quick and dirty one of these, and with a quick > and dirty /etc/rc.d/zfs, it will allow the effect (mostly) having /var > and /usr be from a ZFS pool. > > An unexpected problem was with /var in this configuration, however. For > reasons that are not entirely obvious postfix will not start if /var is > from a ZFS pool. It errors in an unexpected way when trying to start > postfix master. I did not have time to do a full ktrace to see what it > was trying to use, but something is not quite supported, or not > supported in the same way as a FFS. > > An alternative might be to use a zvol with a FFS in it for /var. This > mostly does work as expected from the command line interactively. > However, it won't work from boot because of a couple of problems. > > 1) The names of the devices confuse fsck and it fails on boot. The zvol > will be called something like /dev/zvol/dsk/pool/volname in fstab and > fsck will want to try and clean /dev/zvol/dsk/pool/rvolname which is > wrong for ZFS zvols. You need to use /dev/zvol/rdsk/pool/volname. You > can put a symlink in and it will mostly work, but that is highly manual. > LVM has a symlink tree it creates that points the devices at the right > places in /dev/mapper which
More ZFS testing in 9.0_BETA
I did some more ZFS testing in 9.0_BETA, sorry for the length. There are a lot of advantages and use cases in getting ZFS more completely integrated into the NetBSD universe. Raidframe+LVM is great too, but there are things that are harder to do with it. One of the purposes of these tests was to see how NetBSD ZFS interacted with other ZFS implementations in other OSs. The following was done: "zfs export pool" created from NetBSD and "zfs import pool" into FreeBSD and SmartOS. Both of these worked completely as expected. with full read and write on the pools. There are ZFS features not enabled on pools created with NetBSD, but this does not appear to be a problem. As long as a "zfs upgrade pool" is not performed you can import and export as much as you wish. I have not tried any Linux implementations. Going the other direction did not work quite as expected. On pools created on FreeBSD, the ZFS features that are turned on will not allow the pool to import as writable on NetBSD. This is not really an unexpected thing, given that features on turned on in FreeBSD that are not in NetBSD. The pool will import readonly and appears to work as expected in that case. It should be possible to create pools in FreeBSD with features disabled that will allow a better import. For SmartOS, pools created there will (might?? I only use the filesystems I presented to the ISO) have a GPT header attached to them. When NetBSD sees that, by default, it will create wedges. As a general thing, this wouldn't be a problem. However, "zpool import" does not appear to see dk devices or refuses to deal with them and thus no pools were noticed for import. You can sort of force the issue by creating a directory with symlinks to the /dev/dkN devices in them and then do a "zpool import -d /dev/directory". When this is attempted, the DOMU guest I was using panicked: panic: kernel diagnostic assertion "seg <= BLKIF_MAX_SEGMENTS_PER_REQUEST" failed: file "/usr/src/sys/arch/xen/xen/xbd_xenbus.c", line 1032 I am running an older 8.99.xx with Xen 4.8.x, but as this was from the guest, I suspect that a bug is still present. From the BT, it appears that it tried to do the import, but failed somewhere in the lower level disk routines. I did not try doing this with FreeBSD. Other things of note: Assuming you want to have separate /var and /usr, trying to use ZFS for those is a challenge. One of the biggest issues is that /zbin/zfs and /zbin/zpool have linked against shared libraries in /usr/lib. This is probably not for the best. Further, mostly because of this, you can't reasonably have a /etc/rc.d/zfs start the module and make it usable for critical_filesystems in /etc/rc.conf (you can fake it out by putting the contents of the shared libraries in /usr/lib and then mount over it, but that is a bit ugly). Having /var and /usr use LVM is possible mostly because /etc/rc.d/lvm exists and is started before critical_filesystems. ZFS should probably work the same way. Another item is that mountpoint=legacy really does not work. Right now, there is neither support in the general mount system call for ZFS nor does a /sbin/mount_zfs exist. You can actually get some of this effect by a simple shell script in /sbin called mount_zfs that translates the request into a "/sbin/zfs set mountpoint=XXX" call followed by a "zfs mount XXX". I wrote a quick and dirty one of these, and with a quick and dirty /etc/rc.d/zfs, it will allow the effect (mostly) having /var and /usr be from a ZFS pool. An unexpected problem was with /var in this configuration, however. For reasons that are not entirely obvious postfix will not start if /var is from a ZFS pool. It errors in an unexpected way when trying to start postfix master. I did not have time to do a full ktrace to see what it was trying to use, but something is not quite supported, or not supported in the same way as a FFS. An alternative might be to use a zvol with a FFS in it for /var. This mostly does work as expected from the command line interactively. However, it won't work from boot because of a couple of problems. 1) The names of the devices confuse fsck and it fails on boot. The zvol will be called something like /dev/zvol/dsk/pool/volname in fstab and fsck will want to try and clean /dev/zvol/dsk/pool/rvolname which is wrong for ZFS zvols. You need to use /dev/zvol/rdsk/pool/volname. You can put a symlink in and it will mostly work, but that is highly manual. LVM has a symlink tree it creates that points the devices at the right places in /dev/mapper which allows fsck and mount to work pretty much as needed. We could something like that for ZFS or teach fsck about zvol names. 2) Suppose you wanted to avoid fsck by trying to use wapbl logging. There are missing IOCTLs that will prevent the log directive from working, so "mount -o log /dev/zvol/fsk/pool/volname" will not work. The end result is that while you can define a zvol and put a FFS on it, you can't clean
Re: State of ZFS in 9.0_BETA
m...@netbsd.org writes: > On Sat, Aug 10, 2019 at 09:12:23PM +0100, Chavdar Ivanov wrote: >> I was thinking that if ZFS were in the kernel, it might be easier to >> get ZFS root. > > > You don't need to build ZFS into the kernel to boot it as a root mount > (same for FFS). The code to load modules from the bootloader already > exists. You do, however, need to write/port the functionality to > do a root mount, and teach the bootloader to read ZFS to find the > kernel (unless you use EFI boot, then maybe put the kernel on the FAT > partition). Yes, you will more or less have to do the equivalent of "zfs mount pool/rootfs" (where pool/rootfs has either mountpoint set to / or uses legacy somehow) or "zfs mount -a" (probably not best and it wouldn't work with legacy) and perhaps a "zpool import" before that, if the pools were not imported into this particular system. I note that the FreeBSD folks seem to have a boot loader that does some of this, but I did not look at how it worked too closely. And, of course, you may be able to get ideas from SmartOS. An alternative to messing with the bootloader may be to leverage sysctl init.root and pivot onto a zpool dataset after booting into a root boot partition. I did this with cgd a couple of years ago. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: State of ZFS in 9.0_BETA
On Sat, Aug 10, 2019 at 09:12:23PM +0100, Chavdar Ivanov wrote: > I was thinking that if ZFS were in the kernel, it might be easier to > get ZFS root. You don't need to build ZFS into the kernel to boot it as a root mount (same for FFS). The code to load modules from the bootloader already exists. You do, however, need to write/port the functionality to do a root mount, and teach the bootloader to read ZFS to find the kernel (unless you use EFI boot, then maybe put the kernel on the FAT partition).
