Hi Edward,
Are your zpools on top of iscsi volumes using any kind of RAID levels?
(Mirroring/RAIDZ)?
If not, then iSCSI is behaving exactly the same way as yanking a disk
out. If you yank a disk out of a machine the zpool won't cleanly
"eject", it'll fault and apps will also hang.
On one of our deployments we have two big OpenIndiana iSCSI servers,
exporting targets to a cloud of Solaris 10 hosts. The Solaris 10 hosts
have Zones on top, each zone has it's own zpool which is a mirror of
iSCSI LUNs from the two iSCSI servers.
If one iSCSI server goes away, then after the iSCSI timeout period (180s
I think) all the pools go into a degraded state and stuff keeps on
running. If we stop the iSCSI target on the iSCSI servers then the pools
degrade immediately without waiting for the timeout period, which is great.
If the network fabric goes down, meaning the hosts can't reach both
iSCSI servers, then it does all go to hell and we have to issue hard
reboots of all the host machines to get them back to life. But that's
the nature of centralised storage.
It's worth noting the OpenIndiana iSCSI initiator has a configurable
timeout period. See the iscsiadm man pages.
Cheers,
Alasdair
On 01/10/2012 14:17, Edward Ned Harvey (openindiana) wrote:
From: Mathieu Simon [mailto:[email protected]]
I thought the ZFS mirror on separately hosted iscsi targets was going to be a
great idea ... it turns out it's only great at destroying your pool...
iSCSI is not a shared storage and if you write to the same disk/lun from 2
initiators without "something in between" like DRBD (Linux), AVS, or
HAST (FreeBSD)
things are really expected to cause problems or call other evil dragons. :-)
Understood - But that's not the problem. When I have two OI machines that can both see
the same iscsi targets, they're aware of themselves enough to prevent two systems writing
the disks at the same time. If you have a pool imported on one system, and try to import
it on the other system, it says "currently in use by another machine" or
something. You can only override by force-importing the pool. This is enough to prevent
two systems both writing to the same pool. Each pool will only be imported by one OS at
a time.
The problem is: If one of the targets reboots or becomes unavailable, it's not handled as
removable storage. It's an IO error. And when the system comes back online, it doesn't just
recovery gracefully, resilver, and move onward. It's a perpetual IO error. If the disk comes
online, and maybe you have to "zpool clear" the faulted device, and then it resilvers...
The pool is still unusable. My pool will actually appear to be in a healthy state; "zpool
status" shows a healthy pool, but when an application tries to read/write the pool, I see scsi
IO errors spewing into the syslog. Device timeout occurring, application hanging, and sometimes
the host OS hangs. I have to permanently take the disks offline that were temporarily brought
offline as a result of simply rebooting the iscsi target.
-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175408-4af8ded6
Modify Your Subscription: https://www.listbox.com/member/?&;
Powered by Listbox: http://www.listbox.com
-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be
Modify Your Subscription:
https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4
Powered by Listbox: http://www.listbox.com
