Re: [zfs-discuss] Dedicated server running ESXi with no RAID card, ZFS for storage?
On 2012-11-08 17:49, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: From: zfs-discuss-boun...@opensolaris.org [1] [mailto:zfs-discuss- boun...@opensolaris.org [2]] On Behalf Of Karl Wagner I am just wondering why you export the ZFS system through NFS? I have had much better results (albeit spending more time setting up) using iSCSI. I found that performance was much better, A couple years ago, I tested and benchmarked both configurations on the same system. I found that the performance was equal both ways (which surprised me because I expected NFS to be slower due to FS overhead.) I cannot say if CPU utilization was different - but the IO measurements were the same. At least, indistinguishably different. Based on those findings, I opted to use NFS for several weak reasons. If I wanted to, I could export NFS to more different systems. I know everything nowadays supports iscsi initiation, but it's not as easy to set up as a NFS client. If you want to expand the guest disk, in iscsi, ... I'm not completely sure you *can* expand a zvol, but if you can, you at least have to shut everything down, then expand and bring it all back up and then have the iscsi initiator expand to occupy the new space. But in NFS, the client can simply expand, no hassle. I like being able to look in a filesystem and see the guests listed there as files. Know I could, if I wanted to, copy those things out to any type of storage I wish. Someday, perhaps I'll want to move some guest VM's over to a BTRFS server instead of ZFS. But it would be more difficult with iscsi. For what it's worth, in more recent times, I've opted to use iscsi. And here are the reasons: When you create a guest file in a ZFS filesystem, it doesn't automatically get a refreservation. Which means, if you run out of disk space thanks to snapshots and stuff, the guest OS suddenly can't write to disk, and it's a hard guest crash/failure. Yes you can manually set the refreservation, if you're clever, but it's easy to get wrong. If you create a zvol, by default, it has an appropriately sized refreservation that guarantees the guest will always be able to write to disk. Although I got the same performance using iscsi or NFS with ESXi... I did NOT get the same result using VirtualBox. In Virtualbox, if I use a *.vdi file... The performance is *way* slower than using a *.vmdk wrapper for physical device (zvol). ( using VBoxManage internalcommands createrawvmdk ) The only problem with the zvol / vmdk idea in virtualbox is that every reboot (or remount) the zvol becomes owned by root again. So I have to manually chown the zvol for each guest each time I restart the host. Fair enough, thanks for the info. As I say it was quite a while back and I was using either Xen or KVM (can't remember which). It may be that the performance profiles are/were just very different. I was also just using an old desktop for testing purposes, which skews the performance too (it was far too memory and CPU limited to be used for real). If I was doing this now, I would probably use the ZFS aware OS bare metal, but I still think I would use iSCSI to export the ZVols (mainly due to the ability to use it across a real network, hence allowing guests to be migrated simply) Links: -- [1] mailto:zfs-discuss-boun...@opensolaris.org [2] mailto:boun...@opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Dedicated server running ESXi with no RAID card, ZFS for storage?
On Thu, Nov 08, 2012 at 04:57:21AM +, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: Yes you can, with the help of Dell, install OMSA to get the web interface to manage the PERC. But it's a pain, and there is no equivalent option for most HBA's. Specifcally, on my systems with 3ware, I simply installed the solaris 3ware utility to manage the HBA. Which would not be possible on ESXi. This is important because the systems are in a remote datacenter, and it's the only way to check for red blinking lights on the hard drives. ;-) I thought most IPMI came with full KVM, and also SNMP, and some ssh built-in. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Dedicated server running ESXi with no RAID card, ZFS for storage?
From: Dan Swartzendruber [mailto:dswa...@druber.com] I have to admit Ned's (what do I call you?)idea is interesting. I may give it a try... Yup, officially Edward, most people call me Ned. I contributed to the OI VirtualBox instructions. See here: http://wiki.openindiana.org/oi/VirtualBox Jim's vboxsvc is super powerful - But at first I found it overwhelming, mostly due to unfamiliarity with SMF. One of these days I'm planning to contribute a Quick Start guide to vboxsvc, but for now, if you find it confusing in any way, just ask for help here. (Right Jim?) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Dedicated server running ESXi with no RAID card, ZFS for storage?
From: Karl Wagner [mailto:k...@mouse-hole.com] If I was doing this now, I would probably use the ZFS aware OS bare metal, but I still think I would use iSCSI to export the ZVols (mainly due to the ability to use it across a real network, hence allowing guests to be migrated simply) Yes, if your VM host is some system other than your ZFS baremetal storage server, then exporting the zvol via iscsi is a good choice, or exporting your storage via NFS. Each one has their own pros/cons, and I would personally be biased in favor of iscsi. But if you're going to run the guest VM on the same machine that is the ZFS storage server, there's no need for the iscsi. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Dedicated server running ESXi with no RAID card, ZFS for storage?
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Eugen Leitl On Thu, Nov 08, 2012 at 04:57:21AM +, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: Yes you can, with the help of Dell, install OMSA to get the web interface to manage the PERC. But it's a pain, and there is no equivalent option for most HBA's. Specifcally, on my systems with 3ware, I simply installed the solaris 3ware utility to manage the HBA. Which would not be possible on ESXi. This is important because the systems are in a remote datacenter, and it's the only way to check for red blinking lights on the hard drives. ;-) I thought most IPMI came with full KVM, and also SNMP, and some ssh built- in. Depends. So, one possible scenario: You power up the machine for the first time, you enter ILOM console, you create username password static IP address. From now on, you're able to get the remote console, awesome, great. No need for ipmi-tool in the OS. Another scenario, that I encounter just as often: You inherit some system from the previous admin. They didn't set up IPMI or ILOM. They installed ESXi, and now the only thing you can do is power off the system to do it. But in the situation where I inherit a Linux / Solaris machine from a previous admin who didn't config ipmi... I don't need to power down. I can config the ipmi via ipmi-tools. Going a little further down these trails... If you have a basic IPMI device, then all it does is *true* ipmi, which is a standard protocol. You have to send it ipmi signals via the ipmi-tool command on your laptop (or another server). It doesn't use SSL; it uses either no encryption, or a preshared key. The preshared key is a random HEX 20 character long string. If you configure that at the boot time (as in the first situation mentioned above) then you have to type in at the physical console at first boot: new username, new password, new static IP address etc, and the new encryption key. But if you're running a normal OS, you can skip all that, boot the new OS, and paste all that stuff in via ssh, using the local ipmi-tool to config the local ipmi device. If you have a newer, more powerful ILOM device, then you probably only need to assign an IP address to the ilom. Then you can browse to it via https and do whatever else you need to do. Make sense? Long story short, Depends.;-) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Dedicated server running ESXi with no RAID card, ZFS for storage?
On 2012-11-09 16:14, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: From: Karl Wagner [mailto:k...@mouse-hole.com] If I was doing this now, I would probably use the ZFS aware OS bare metal, but I still think I would use iSCSI to export the ZVols (mainly due to the ability to use it across a real network, hence allowing guests to be migrated simply) Yes, if your VM host is some system other than your ZFS baremetal storage server, then exporting the zvol via iscsi is a good choice, or exporting your storage via NFS. Each one has their own pros/cons, and I would personally be biased in favor of iscsi. But if you're going to run the guest VM on the same machine that is the ZFS storage server, there's no need for the iscsi. Well, since the ease of re-attachment of VM hosts to iSCSI was mentioned a few times in this thread (and there are particular nuances with iSCSI to localhost), it is worth mentioning that NFS files can be re-attached just as easily - including the localhost. Cloning disks is just as easy when they are zvols or files in dedicated datasets; note that disk image UUIDs must be re-forged anyway (see doc). Also note, that in general, there might be need for some fencing (i.e. only one host tries to start up a VM from a particular backend image). I am not sure iSCSI inherently does a better job than NFS at this?.. //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Dedicated server running ESXi with no RAID card, ZFS for storage?
On 2012-11-09 16:11, Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) wrote: From: Dan Swartzendruber [mailto:dswa...@druber.com] I have to admit Ned's (what do I call you?)idea is interesting. I may give it a try... Yup, officially Edward, most people call me Ned. I contributed to the OI VirtualBox instructions. See here: http://wiki.openindiana.org/oi/VirtualBox Jim's vboxsvc is super powerful Thanks for kudos, and I'd also welcome some on the SourceForge project page :) http://sourceforge.net/projects/vboxsvc/ for now, if you find it confusing in any way, just ask for help here. (Right Jim?) I'd prefer the questions and discussion on vboxsvc to continue in the VirtualBox forum, so it's all in one place for other users too. It is certainly an offtopic for the lists about ZFS, so I won't take this podium for too long :) https://forums.virtualbox.org/viewtopic.php?f=11t=33249 One of these days I'm planning to contribute a Quick Start guide to vboxsvc, I agree that the README might need cleaning up, so far it is like a snowball growing with details and new features. Perhaps some part should be separated into a concise quick-start guide that would not scare people off by the sheer amount of letters ;) I don't think I can point to a chapter and say Take this as the QuickStart :( - But at first I found it overwhelming, mostly due to unfamiliarity with SMF. The current README does, however, provide an overview of SMF as was needed by some of the inquiring users, and an example on command-line creation of a service to wrap a VM. A feature to do this by the script itself is pending, somewhat indefinitely. Also note that for OI desktop users in particular (and likely for other OpenSolaris-based OSes with X11 too), I'm now adding features to ease management of VMs that are not executed headless, but rather are interactive. Now these can too be wrapped as SMF services to automate shutdown and/or backgrounding into headless mode and back. I made and use this myself to enter other OSes on my laptop that are dual-bootable and can run in VBox as well as on hardware. There is also a new foregrounding startgui mode that can trap the signals which stop its terminal, and properly savestate or shutdown the VM, as well as this wraps taking of ZFS snapshots for VM disk resources, if applicable. There is also a mode where this spawns a dedicated xterm for the script's execution; by closing the xterm you can properly stop the VM with the preselected method of your choice with one click, before you log out of X11 session. However, this part of my work was almost in vain - the end of X11 session happens as a bruteforce close of X-connections, so the interactive GUIs just die before they can process any signals. This makes sense for networked X-servers that can't really send signals to remote client OSes, but is rather stupid for local OS. I hope the desktop environment gurus might come up with something. Or perhaps I'll come up with an SMF wrapper for X sessions that the vbox startgui feature could depend on, and the close of a session would be an SMF disablement. Hopefully, spawned local X-clients would also be under the SMF contract and would get chances to stop properly :) Anyway, if anybody else is interested in the new features described above - check out the code repository for the vboxsvc project (this is not yet so finished as to publish a new package version): http://vboxsvc.svn.sourceforge.net/viewvc/vboxsvc/lib/svc/method/vbox.sh http://vboxsvc.svn.sourceforge.net/viewvc/vboxsvc/var/svc/manifest/site/vbox-svc.xml http://vboxsvc.svn.sourceforge.net/viewvc/vboxsvc/usr/share/doc/vboxsvc/README-vboxsvc.txt See you in the VirtualBox forum thread if you do have questions :) //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Forcing ZFS options
There are times when ZFS options can not be applied at the moment, i.e. changing desired mountpoints of active filesystems (or setting a mountpoint over a filesystem location that is currently not empty). Such attempts now bail out with messages like: cannot unmount '/var/adm': Device busy cannot mount '/export': directory is not empty and such. Is it possible to force the new values to be saved into ZFS dataset properties, so they do take effect upon next pool import? I currently work around the harder of such situations with a reboot into a different boot environment or even into a livecd/failsafe, just so that the needed datasets or paths won't be busy and so I can set, verify and apply these mountpoint values. This is not a convenient way to do things :) Thanks, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Forcing ZFS options
Do you move the pools between machines, or just on the same physical machine? Could you just use symlinks from the new root to the old root so that the names work until you can reboot? It might be more practical to always use symlinks if you do a lot of moving things around, and then you wouldn't have to figure out how to do the reboot shuffle. Instead, you could just shuffle the symlinks. Gregg Wonderly On Nov 9, 2012, at 10:47 AM, Jim Klimov jimkli...@cos.ru wrote: There are times when ZFS options can not be applied at the moment, i.e. changing desired mountpoints of active filesystems (or setting a mountpoint over a filesystem location that is currently not empty). Such attempts now bail out with messages like: cannot unmount '/var/adm': Device busy cannot mount '/export': directory is not empty and such. Is it possible to force the new values to be saved into ZFS dataset properties, so they do take effect upon next pool import? I currently work around the harder of such situations with a reboot into a different boot environment or even into a livecd/failsafe, just so that the needed datasets or paths won't be busy and so I can set, verify and apply these mountpoint values. This is not a convenient way to do things :) Thanks, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Forcing ZFS options
On 2012-11-09 18:06, Gregg Wonderly wrote: Do you move the pools between machines, or just on the same physical machine? Could you just use symlinks from the new root to the old root so that the names work until you can reboot? It might be more practical to always use symlinks if you do a lot of moving things around, and then you wouldn't have to figure out how to do the reboot shuffle. Instead, you could just shuffle the symlinks. No, this concerns datasets within the machine. And symlinks often don't cut it. For example, I've recently needed to switch '/var' from an automounted filesystem dataset into a legacy one with the mount from /etc/vfstab. I can't set the different mountpoint value (legacy) while the OS is up and using the 'var', and I don't seem to have a way to do this during reboot automatically (short of crafting an SMF script that would fire early in the boot sequence - but that's a workaround outside ZFS technology, as is using the livecd or a failsafe boot or another BE). A different example is that sometimes uncareful works with beadm leave the root dataset with a non='/' mountpoint attribute. While the proper rootfs is forced to mount at the root node, it is not clean to have the discrepancy. However, I can not successfully zfs set mountpoint=/ rpool/ROOT/bename while booted into this BE. Forcing the attribute to save the value I need, so it takes effect after reboot - that's what I am asking for (if that was not clear from my first post). Thanks, //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] IBMsdd to MPXIO
I have zpools now that consist of vpaths. I'm exporting the zpool and want to import on a server using mpxio. Will zfs care or will it see the zpool although the multipathing driver has changed? I'm assuming not.___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
