On Wed, Aug 04, 2010 at 10:54:48AM -0700, Mike Vallaly wrote:
> Sorry for the lateness in my reply. Just stumbled across this
> thread.. ;)
>
> Part of the problem with MPIO in linux with two (or more) interfaces
> connected to the same Ethernet segment is "arp flux". Essentially all
> traffic will by default only exit out one path (mac address) on a
> multi-homed network. The fix for this is to explicitly tie the
> interface to a route rule which ensures traffic leaves via the
> interface the application intended.
>
> Here is the script we use in Debian to properly set the interface
> routes for MPIO with equallogic. (IE: /etc/scripts/san-interface)
>
Uhm, why not just use open-iscsi 'ifaces' feature to bind each
iscsi iface/session to specific ethernet interface?
-- Pasi
> <snip>
> #!/bin/bash
>
> # Michael Vallaly (Feb 2008)
>
> # This script configures local interfaces for use CNU's MP-IO iSCSI
> SAN Network
> # NOTE: This script may be called from /etc/network/interfaces without
> parameters
>
> SAN_NETWORKS="10.99.99.0/24"
> SAN_MTU="9000"
>
> IP_BIN="/bin/ip"
> ETHTOOL_BIN="/usr/sbin/ethtool"
>
> ########################################################################################
>
> # Check for required binaries
> for req_bin in $IP_BIN $ETHTOOL_BIN; do
> if [ ! -x "$req_bin" ]; then
> echo "Can't execute ${req_bin}! Aborting.."
> exit 1
> fi
> done
>
> usage="Usage: $0 -i <interface> -m <add/del>"
>
> while getopts "i:m:" options; do
> case $options in
> i ) interfaces+=" $OPTARG";;
> m ) action=$OPTARG;;
> \? ) echo $usage
> exit 1;;
> * ) echo $usage
> exit 1;;
> esac
> done
>
> # Check for ifup/down enviornment variables
> if [[ -n $MODE && -n $IFACE ]]; then
> interfaces=$IFACE
> action=$MODE
> fi
>
> # Figure out what we are doing
> case $action in
> start ) action="add";;
> add ) action="add";;
> stop ) action="del";;
> del ) action="del";;
> * ) echo $usage
> exit 1;;
> esac
>
> for interface in $interfaces; do
>
> # Check that the interface exists before we go playing with it
> if ! ($IP_BIN addr |egrep -nqe "inet.*$interface" && $IP_BIN link |
> egrep -nqe "$interface.*,UP"); then
> continue
> fi
> table_num=$((`echo ${interface} |tr -d [[:alpha:]]` + 10))
> interface_ip=`$IP_BIN route show scope link proto kernel dev $
> {interface} |awk '{print $3}'`
>
> if [ $table_num -gt 252 ]; then
> echo "Invalid SAN interface (${table_num}) specified!"
> exit 1
> fi
>
> for network in $SAN_NETWORKS; do
>
> # Configure our remote SAN networks
>
> localnet=`$IP_BIN route |grep "${interface} proto kernel" |cut -
> d" " -f1`
> existing_san_iface=`$IP_BIN route show ${network} |grep -we "via" |
> awk '{print $5}'`
>
> # Don't add networks if they are locally connected
> if [[ "$localnet" == "$network" ]]; then
> continue
> else
>
> # Set our default gateway for remote networks
> local=`echo $localnet |cut -d. -f1-3`
>
> if [[ "$action" == "add" ]]; then
>
> # Create a unique route table
> $IP_BIN route add ${localnet} dev ${interface} table $
> {table_num}
> $IP_BIN route add ${network} via ${local}.1 dev ${interface}
> table ${table_num}
> $IP_BIN rule add from ${interface_ip}/32 lookup ${table_num}
> route_match=`echo $existing_san_iface $interface |tr -t ' '
> '\n'|sort -u`
>
> else
>
> # Delete the route table
> $IP_BIN rule del from ${interface_ip}/32 lookup ${table_num}
> 2> /dev/null
> route_match=`echo $existing_san_iface $interface |tr -t ' '
> '\n'|sort -u |grep -v ${interface}`
>
> fi
>
> # Generate required next hops
> route_opt=""
> for dev in $route_match; do
> route_opt="$route_opt nexthop via ${local}.1 dev ${dev}"
> done
>
> # Cleanup default route
> $IP_BIN route del ${network} via ${local}.1 2> /dev/null
>
> # Add/ReAdd the default route
> if [ "${route_opt}" != "" ]; then
> eval $IP_BIN route add ${network} scope global ${route_opt}
> fi
>
> fi
>
> done
>
> # Flush the routing cache
> $IP_BIN route flush cache
>
> # Configure our local network interfaces
> # Configure our local network interfaces
>
> if [[ "$action" == "add" ]]; then
>
> # Set the proper MTU for the network interface (note this may take
> the interface offline!)
> if [ "$($IP_BIN link show $interface |grep "mtu" |cut -d" " -f
> 5)" != "9000" ]; then
> $IP_BIN link set $interface mtu $SAN_MTU
> fi
>
> # Force flowcontrol on
> $ETHTOOL_BIN --pause $interface autoneg off rx on tx on
>
> # Only ARP for local interface
> echo "1" > /proc/sys/net/ipv4/conf/${interface}/arp_ignore
>
> else
>
> # Set the proper MTU for the network interface (note this may take
> the interface offline!)
> if [ "$($IP_BIN link show $interface |grep "mtu" |cut -d" " -f 5)"
> == "9000" ]; then
> $IP_BIN link set $interface mtu 1500
> fi
>
> # Force flowcontrol autoneg
> $ETHTOOL_BIN --pause $interface autoneg on
>
> # ARP for any interface
> echo "0" > /proc/sys/net/ipv4/conf/${interface}/arp_ignore
>
> fi
>
> done
>
> </snip>
>
> You can then add something similar to this in your /etc/network/
> interface file:
>
> <snip>
> # The First SAN interface
> iface eth1 inet static
> address 10.99.99.100
> netmask 255.255.255.0
> post-up /etc/scripts/san-interface
> pre-down /etc/scripts/san-interface
>
> # The Second SAN interface
> iface eth2 inet static
> address 10.99.99.101
> netmask 255.255.255.0
> post-up /etc/scripts/san-interface
> pre-down /etc/scripts/san-interface
>
> </snip>
>
> Cheers..
>
> -Mike
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "open-iscsi" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/open-iscsi?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/open-iscsi?hl=en.
