> From: Alasdair Lumsden [mailto:[email protected]]
>
> Are your zpools on top of iscsi volumes using any kind of RAID levels?
> (Mirroring/RAIDZ)?
Here's the exact process I used to set it up, and the exact final config: (I
documented it as I went along, and this is pasted.)
(Short answer, mirroring, local iscsi target and remote iscsi target)
Each system will export via iscsi, all four disks: c4t2d0 c4t3d0 c4t4d0 c4t5d0
Each system will initiate a connection to all eight disks, including the ones
at 127.0.0.1
When creating zpool, don't use the local device name (c4t2d0 or whatever)
because that's not available to the other system.
Use only the iscsi target device names.
vmhost1 will create the "HAstorage1" pool, with (local c4t2d0 mirrored to
remote c4t4d0) and (local c4t3d0 mirrored to remote c4t5d0)
vmhost2 will create the "HAstorage2" pool, with (local c4t2d0 mirrored to
remote c4t4d0) and (local c4t3d0 mirrored to remote c4t5d0)
Most of the time, HAstorage1 will be imported by vmhost1
Most of the time, HAstorage2 will be imported by vmhost2
When you want to, you can export HAstorage2 on vmhost2, and then import it on
vmhost1. (And vice-versa)
Or, if one system dies ungracefully, the other one can force-import the pool.
Enough talk. Let's do it:
Do this on both systems, to enable targets:
sudo pkg install pkg:/network/iscsi/target
sudo svcadm enable -s svc:/system/stmf
sudo svcadm enable -s svc:/network/iscsi/target
Do this on both systems, to enable initiators:
sudo iscsiadm modify discovery --static enable
On both machines:
for DISKNUM in 2 3 4 5 ; do sudo sbdadm create-lu /dev/rdsk/c4t${DISKNUM}d0 ;
done
for GUID in `sudo sbdadm list-lu | grep rdsk | sed 's/ .*//'` ; do sudo stmfadm
add-view $GUID ; done
sudo itadm create-target
>From the generated target names, and IP addresses, create the following two
>commands, which will need to both be executed on both machines ... But only
>one at a time. So you can keep track of which disks are on which machines.
sudo iscsiadm add static-config
iqn.2010-09.org.openindiana:02:78e7f2a1-1f49-4dfa-e66e-e0cdcc2cfdfe,192.168.3.10
sudo iscsiadm add static-config
iqn.2010-09.org.openindiana:02:0beb4f34-b30e-ec64-a9df-9790c8e89e54,192.168.3.11
sudo format -e
Make a note of the new device names. And hit Ctrl-C
Create the following information:
(H1 disks are on Host 1)
export H1T0=c5t600144F0080027B01DE3506793070004d0
export H1T1=c5t600144F0080027B01DE3506793060001d0
export H1T2=c5t600144F0080027B01DE3506793060002d0
export H1T3=c5t600144F0080027B01DE3506793060003d0
(H2 disks are on Host 2)
export H2T0=c5t600144F0080027B907F8506793090004d0
export H2T1=c5t600144F0080027B907F8506793080001d0
export H2T2=c5t600144F0080027B907F8506793080002d0
export H2T3=c5t600144F0080027B907F8506793090003d0
Only do this on one server:
sudo zpool create HAstorage1 mirror $H1T0 $H2T0 mirror $H1T1 $H2T1
sudo zpool create HAstorage2 mirror $H1T2 $H2T2 mirror $H1T3 $H2T3
Now, if you want to pass the pool over the other system, just export & import.
> On one of our deployments we have two big OpenIndiana iSCSI servers,
> exporting targets to a cloud of Solaris 10 hosts. The Solaris 10 hosts
> have Zones on top, each zone has it's own zpool which is a mirror of
> iSCSI LUNs from the two iSCSI servers.
Based on what I'm experiencing here, I recommend you perform rituals to ensure
neither of the OI iscsi servers will never, ever go down. Don't apply system
updates. Don't reboot. Monitor the syslogs of all the systems for any SCSI
errors.
If one of the servers *does* reboot, you may or may not show faulted disks in
the clients. If you see faulted disks, then online'ing them and clearing them
will only make zpool stop complaining about faulted disks - it will not make
the scsi errors go away. You'll be in a bad state.
The only thing that seems to solve the problem is to detach the faulted device
from the client, attach a new one. (Which is of course, the same disk as
before.)
> If one iSCSI server goes away, then after the iSCSI timeout period (180s
> I think) all the pools go into a degraded state and stuff keeps on
> running. If we stop the iSCSI target on the iSCSI servers then the pools
> degrade immediately without waiting for the timeout period, which is great.
>
> If the network fabric goes down, meaning the hosts can't reach both
> iSCSI servers, then it does all go to hell and we have to issue hard
> reboots of all the host machines to get them back to life. But that's
> the nature of centralised storage.
>
> It's worth noting the OpenIndiana iSCSI initiator has a configurable
> timeout period. See the iscsiadm man pages.
What's different in your setup from mine? Why doesn't your stuff fail the way
I'm seeing?
I'm going to dig more into this. Partly because I really *want* it to work,
and partly because the alternatives suck, and partly because you say yours is
working.
As part of digging into this more, I actually rebuilt both servers from
scratch, documenting the whole process step-by-step, on Saturday, and
replicating all the data to all the pools on Sunday. I'll be free to work on
it some more (cause failures and see what I can learn about them) tomorrow.
-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be
Modify Your Subscription:
https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4
Powered by Listbox: http://www.listbox.com
