I should have better explained why this script is useful. Tardiness
should be expected with me ;)
We have a setup something like this:
LunX <-> Openiscsi <-> eth2 (10.99.99.101) <-> IscsiTarget
(10.99.99.10)
and
LunX <-> Openiscsi <-> eth3 (10.99.99.102) <-> IscsiTarget
(10.99.99.10)
Both (eth2) 10.99.99.101 and (eth3) 10.99.99.102 are on a single Linux
Host and we have a single IscsiTarget on the same network
(10.99.99.0/24) which we create two iscsi TCP sessions to
(10.99.99.10).
What happens is you get a linux routing table which looks something
like this:
<snip>
mvall...@darkstar:~$ ip ro sh
10.99.99.0/24 dev eth2 proto kernel scope link src 10.99.99.101
10.99.99.0/24 dev eth3 proto kernel scope link src 10.99.99.102
default via 10.99.99.1 dev eth2 proto static
</snip>
Connectivity _WILL_ work, but not likely as one would expect. I will
attempt to enumerate the unexpected (land-mine) behavior below:
#1. Arp Flux - In Linux the MAC address by default is tied to the HOST
and not the NIC (counter intuitive I know), meaning _ANY_ active
network interface on a host _MAY_ reply to an ARP request for _ANY_
MAC address the host has NICs for. What this means is that you have a
race condition when two NICs are placed on the same network, as either
of them CAN and WILL respond to ARP requests for the other, resulting
in (my personal experience) unbalanced inbound traffic across two NICs
on the same network.
#2. Routing Table - In the above routing table snippet, you will
notice we have two network routes for 10.99.99.0/24 (one attached to
eth2 and one attached to eth3) the HOST can use when contacting the
IscsiTarget (10.99.99.10). The land-mine here is that even with two
separate processes (separate open-iscsi sessions), both sessions will
use the most specific match in the routing table (IE: "10.99.99.0/24
dev eth2 proto kernel scope link src 10.99.99.101") and traffic
will exit only via eth2. To make matters even worse we have a standard
default gateway of 10.99.99.1, which works until the eth2 network link
is severed, at which point we lose all ability to get packets off the
10.99.99.0/24 network despite sill having a working eth3.
So the script I attached earlier fixes these issues,
First it changes the ARP function on the SAN interfaces. Specifically
'echo "1" > /proc/sys/net/ipv4/conf/${interface}/arp_ignore' on the
SAN interfaces to prevent the Linux kernel from responding to ARPs on
interfaces other than via the NIC containing the MAC used in the ARP
response.
Second it configures iproute2 route rules, used in conjunction to the
standard linux routing table, which forces traffic leaving from the
HOST, to exit using the interface the IP packets are sourced sourced
from.
Third it configures a multi-hop default route for the SAN networks,
which provides dead gateway detection and fail-over should one of the
interfaces on the 10.99.99.0/24 network fail.
Hopefully this helps.. (or explains my maddness)
-Mike
On Aug 5, 1:32 am, "Ulrich Windl" <[email protected]>
wrote:
> >>> Mike Vallaly <[email protected]> schrieb am 04.08.2010 um 19:54 in
>
> Nachricht
> <a5c39a25-6de0-4f00-8c0b-79186c6f8...@f33g2000yqe.googlegroups.com>:
>
> > Sorry for the lateness in my reply. Just stumbled across this
> > thread.. ;)
>
> > Part of the problem with MPIO in linux with two (or more) interfaces
> > connected to the same Ethernet segment is "arp flux". Essentially all
> > traffic will by default only exit out one path (mac address) on a
> > multi-homed network. The fix for this is to explicitly tie the
> > interface to a route rule which ensures traffic leaves via the
> > interface the application intended.
>
> Mike,
>
> looking at the script I tried to figure out what the script actually does.
> Can you describe in more detail what the problem is, and how the script fixes
> it?
>
> Regards,
> Ulrich
> [...]
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/open-iscsi?hl=en.
