> From: Edward Ned Harvey (lopser) [mailto:[email protected]] > Sent: Monday, October 01, 2012 12:18 PM > > When creating zpool, don't use the local device name (c4t2d0 or whatever) > because that's not available to the other system. > Use only the iscsi target device names.
I am revisiting this issue today. I've tried everything I can think of to recreate this issue, and haven't been able to do it. I have certainly encountered some bad behaviors - which I'll expound upon momentarily - but they all seem to be addressable, fixable, logical problems, and none of them result in a supposedly good pool (as reported in zpool status) returning scsi IO errors or halting the system. The most likely explanation right now, for the bad behavior I saw before, perpetual IO error even after restoring connection, is that I screwed something up in my iscsi config the first time. In fact, I suspect, the most likely error I made is the one mentioned above. Only use the iscsi multipath device names, not the local device names. I learned that the hard way, wrote down this comment, attempted to rebuild/recreate, thought I succeeded, and perhaps didn't. In any event, I've rebuilt the servers both from scratch now, and definitely cleaned up any possible errors in my iscsi config that might have previously existed. Now, the state of the world is: I have 4 disks on each machine. They are all iscsi targets. Each machine connects to itself and the other machine, so each machine can see 8 disks (4 local, 4 remote.) Create a pool that mirrors local disks against remote disks. Only import the pool on one server. Rest assured, if the one server dies, you could force import using the other system, if necessary. Herein lie the new problems: If I don't export the pool before rebooting, then either the iscsi target or initiator is shutdown before the filesystems are unmounted. So the system spews all sorts of error messages while trying to go down, but it eventually succeeds. It's somewhat important to know if it was the target or initiator that went down first - If it was the target, then only the local disks became inaccessible, but if it was the intiiator, then both the local and remote disks became inaccessible. I don't know yet. Upon reboot, the pool fails to import, so the svc:/system/filesystem/local service fails, and comes up in maintenance mode. The whole world is a mess, you have to login at physical text console to export the pool, and reboot. But it comes up cleanly the second time. These sorts of problems seem like they should be solvable by introducing some service manifest dependencies... But there's no way to make it a generalization for the distribution as a whole (illumos/openindiana/oracle). It's just something that should be solvable on a case-by-case basis. If you are going to be an initiator only, then it makes sense for svc:/network/iscsi/initiator to be required by svc:/system/filesystem/local If you are going to be a target only, then it makes sense for svc:/system/filesystem/local to be required by svc:/network/iscsi/target If you are going to be a target & initiator, then you could get yourself into a deadlock situation. Make the filesystem depend on the initiator, and make the initiator depend on the target, and make the target depend on the filesystem. Uh-oh. But we can break that cycle easy enough in a lot of situations - If you're doing as I'm doing, where the only targets are raw devices (not zvols) then it should be ok to make the filesystem depend on the initiator, which depends on the target, and the target doesn't depend on anything. If you're both a target and an initiator, but all of your targets are zvols that you export to other systems (you're not nesting a filesystem in a zvol of your own, are you?) then it's ok to let the target needs filesystem and filesystem needs initiator, but initiator doesn't need anything. So in my case, I'm sharing raw disks, I'm going to try and make filessytem needs initiator, initiator needs target, and target doesn't need anything. Haven't tried yet ... Hopefully google will help accelerate me figuring out how to do that. ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
