Thanks for the response Mike.  I've went through so many changes over
the last week and the environment has pretty much been peeled back to
the core.  We're just now building everything back on.  If we run into
the issue again, I'll be sure to check the logs for more specifics as
well as pay attention to those tunables in iscsid.conf.

-----Original Message-----
From: []
On Behalf Of Mike Christie
Sent: Thursday, July 09, 2009 2:49 AM
Subject: Re: iscsiadm -m iface + routing

hootjr29 wrote:
> Hi all,
> I'm currently attempting to implement a Dell EqualLogic iSCSI solution
> connected through m1000e switches to Dell m610 blades with (2) iSCSI
> dedicated nics in each blade running Oracle VM Server v2.1.5 (which, I
> believe, is based off of RHEL5.1).
> [r...@oim6102501 log]# rpm -qa | grep iscsi
> iscsi-initiator-utils-
> [r...@oim6102501 log]# uname -a
> Linux oim6102501 2.6.18- #1 SMP Tue May 12 19:21:30
> EDT 2009 i686 i686 i386 GNU/Linux
> I have setup a bond with the two nics to test this out initially and
> had no problems.  This allows for failover (active-passive bond).  I
> discovered my targets, I logged into sessions, and recognized
> everything from dm-multipath.  I fdisked, formatted drives, and
> mounted them.    I then ran lots of dd's to and from the disks with
> speeds around 90MB/sec when `dd if=/dev/mapper/ovm-1-lun0p1 of=/dev/
> null bs=1M count=1000`
> So since this is going to be one of many VM servers in our OVM cluster
> with multiple VM's running on it (of which many are database servers),
> I wanted to try to make this more efficient.  Therefore, I read that
> by using the `iscsiadm -m iface` syntax you can (instead of bonding)
> setup (2) nics individually, each with an IP on that same segment as
> your iSCSI storage.  From what I understand, this allows (2) sessions
> to be created to each volume-- which should give you a little more
> throughput.  I did this:
> iscsiadm -m iface -I eth2 --op=new
> iscsiadm -m iface -I eth3 --op=new
> iscsiadm -m iface -I eth2 --op=update -n iface.hwaddress -v
> 00:10:18:3A:5B:6C iscsiadm -m iface -I eth3 --op=update -n
> iface.hwaddress -v 00:10:18:3A:5B:6E iscsiadm -m discovery -t st -p
> -P 1 iscsiadm -m node --loginall=all
> dm-multipath sees 2 paths now to each volume.  If I run `iscsiadm -m
> session -P 3` I can see which /dev/sdX device is used by which
> multipath device.  I have multipath setup to load balance across both
> paths with rr_min_io set to 10 in /etc/multipath.conf (which it is my
> understanding that this will send 10 I/Os to one path and then switch
> to the other path).
> my eth2 is
> my eth3 is
> my group is
> my eql interface#1 is
> my eql interface#1 is
> [r...@oim6102501 log]# netstat -rn
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags   MSS Window
> irtt Iface
>   U         0 0
> 0 vlan10
>     U         0 0
> 0 eth3
>     U         0 0
> 0 eth2
>     U         0 0
> 0 eth3
>         UG        0 0
> 0 vlan10
> ========
> I'm now having some odd issues where everything appears to work 
> I can mount drives and dd stuff around.  But then I will occasionally
> get "Reset received on the connection" from my EqualLogic logs.  I
> will see the same thing in /var/log/messages from my kernel scsi layer
> as well as dm-multipath layer.  When I look at `iscsiadm -m session -P
> 2` I can see the following:
> iSCSI Connection State: TRANSPORT WAIT
> iSCSI Session State: Unknown
> Internal iscsid Session State: REPOEN
> By the way, is that a bug?  "REPOEN"?  should it be "REOPEN"?

Yes. I put up a fix on

> Within about 1-2 minutes it will reconnect.  But I'm a bit baffled
> what would cause this.

In /var/log/messages do you see something about a nop or ping timing 
out, or do you just see something about a host reset succeeding?

The initiator sends a iscsi ping (nop iscsi command) to the target every

node.conn[0].timeo.noop_out_interval. If we do not get a response in 
node.conn[0].timeo.noop_out_timeout then we will drop the connection and


If we reconnected within a couple of seconds, then it might have been 
the driver being too agressive and you may want to increase those nops 
values. If it is taking a couple minutes to reconnect then maybe 
something is wrong with the network temporarily.

If you are only seeing the host reset succeed message then you are 
probably sending more IO than the target device can handle. You should 
try lowering the  node.session.cmds_max and node.session.queue_depth.

> =========
> Since I am creating two iscsi sessions (one out eth2 and eth3), I'm
> wondering how routing plays into sessions.  Since iscsiadm is given
> the hwaddress, does iscsid need to care much about routing?  In other
> words, let's say that, for whatever reason, my session that I had
> through eth3 (that session, by the way, is connected to
> on the EqualLogic) timesout.  iscsid sees this and attempts to REOPEN
> the session.  Since my routing table shows eth2 above the route for
> eth3, IP-wise, eth2 will be the interface that would typically be
> chosen for that traffic to route out of.  However, is iscsid smart
> enough to not use that route and instead select the iface (based on
> hwaddress) to use for that reconnection?

When you use the iface binding iscsid is going to tell the network layer

to ignore the route tables and bind the session to whatever nic you 
specified in the iface/target binding.

You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To post to this group, send email to
To unsubscribe from this group, send email to
For more options, visit this group at

Reply via email to