Just to follow up on this, using RHEL5 and OFED 1.2 rc4, I was able
to do enough rudimentary testing to convince myself that IB
bonding was working. I was able to use ib-bond, as well
as the use of the openib.conf file to enable bonding on startup,
including both(separately) IPOIBBOND_ENABLE and IPOIBHA_ENABLE.
One thing I was not able to do however, was to start IB bonding
using the standard bonding modifications to /etc/modprobe.conf
and /etc/sysconfig/network-scripts/ifcfg* files. Should this be possible,
and are there perhaps some required settings I am missing? I'll
include my file modifications and some output below.
modprobe.conf:
alias bond0 bonding
options bond0 mode=active-backup miimon=100
ifcfg-bond0:
DEVICE=bond0
IPADDR="172.22.0.23"
NETMASK="255.255.0.0"
NETWORK="172.22.0.0"
BROADCAST="172.22.255.255"
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_SLAVE0=ib0
BONDING_SLAVE0=ib1
ifcfg-ib0:
DEVICE=ib0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
ifcfg-ib1:
DEVICE=ib1
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
[EMAIL PROTECTED] ~]# ifconfig
bond0 Link encap:InfiniBand HWaddr
80:00:04:05:FE:80:00:00:00:00:00:00:00:0
0:00:00:00:00:00:00
inet addr:172.22.0.23 Bcast:172.22.255.255 Mask:255.255.0.0
UP BROADCAST MASTER MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:1352 (1.3 KiB)
dmesg:
...
Ethernet Channel Bonding Driver: v3.1.1 (September 26, 2006)
bonding: MII link monitoring set to 100 ms
ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave ib0.
bonding: bond0: Warning: enslaved VLAN challenged slave ib0. Adding
VLANs will b
e blocked as long as ib0 is part of bond bond0
bonding: bond0: Warning: The first slave device you specified does not
support s
etting the MAC address. This bond MAC address would be that of the
active slave.
ADDRCONF(NETDEV_UP): ib0: link is not ready
bonding: bond0: Warning: failed to get speed and duplex from ib0,
assumed to be
100Mb/sec and Full.
bonding: bond0: making interface ib0 the new active one.
bondingbond_send_grat_arp: bond bond0 slave ib0
bonding: bond0: first active interface up!
bonding: bond0: enslaving ib0 as an active interface with an up link.
bonding: bond0: Adding slave ib1.
bonding: bond0: Warning: enslaved VLAN challenged slave ib1. Adding
VLANs will b
e blocked as long as ib1 is part of bond bond0
ADDRCONF(NETDEV_UP): ib1: link is not ready
bonding: bond0: Warning: failed to get speed and duplex from ib1,
assumed to be
100Mb/sec and Full.
bonding: bond0: enslaving ib1 as a backup interface with an up link.
ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
...
bonding: bond0: Interface ib0 is already enslaved!
ib0: enabling connected mode will cause multicast packet drops
ib0: mtu > 2044 will cause multicast packet drops.
bonding: bond0: link status definitely down for interface ib0, disabling it
bonding: bond0: making interface ib1 the new active one.
bondingbond_send_grat_arp: bond bond0 slave ib1
bonding: bond0: Interface ib1 is already enslaved!
ib1: enabling connected mode will cause multicast packet drops
ib1: mtu > 2044 will cause multicast packet drops.
bonding: bond0: link status definitely down for interface ib1, disabling it
bondingbond_send_grat_arp: bond bond0 slave NULL
bonding: bond0: now running without any active interface !
Thanks,
Bob
Scott Weitzenkamp (sweitzen) wrote:
Bob, it is now possible to configure IPoIB bonding in
/etc/infiniband/openib.conf, this configuration file includes the
following boilerplate.
# Enable the bonding driver on startup
IPOIBBOND_ENABLE=no
# Set bond interface names
#IPOIB_BONDS=bond0,bond1
# Set specific bond params; address and slaves
#bond0_IP=10.10.10.1
#bond0_SLAVES=ib0,ib1
#bond1_IP=20.10.10.1
#bond1_SLAVES=ib2,ib3,ib4
Scott Weitzenkamp
SQA and Release Manager
Server Virtualization Business Unit
Cisco Systems
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Or Gerlitz
Sent: Tuesday, May 29, 2007 12:56 AM
To: Bob Kossey
Cc: OpenFabrics General
Subject: [ofa-general] Re: ipoib / bonding and OFED
Bob Kossey wrote:
I copied OR since I think this is related to his OFED HA work, and
he might have some insights. A few more questions for Or:
I was trying to use ipoib bonding with OFED 1.2 rc2 and a
2.6.9 kernel,
but was not able to get it to work so far. I saw your
Sonoma bonding
slides, and you mention kernel bonding driver changes were needed.
2. Is there a minimum kernel version, with the kernel bonding driver
changes, that is required to use bonding with OFED ipoib?
Just to have a base line here: to get bonding to work with IPoIB, you
should use the bonding driver provided with OFED 1.2. This
driver is the
upstream one (of 2.6.20) being patched to support IPoIB and
backported
to RH5, SLES10 and RH4 U3/4/5, other kernels are not supported.
If you were using the ofed bonding on a system that matches
the support
matrix it should worl. If do have problems under this config, please
either open a bug at the ofed bugzilla
@ bugs.openfabrics.org assigned to [EMAIL PROTECTED] (Moni Shoua) or
send first report/question to Moni and CC [EMAIL PROTECTED]
Please note that between RC2 and RC4 (to be released today etc) some
bugs were fixed, you can search in the bugzilla to see what.
3. The bonding driver uses the HWADDR from the underlying ipoib
devices, how does it obtain the HWADDR? Does it use the
full 20 bytes,
or some subset?
when enslaving IPoIB devices, the bonding driver uses the full hw
address of the active slave, it simply looks on the dev_addr field of
the slave struct netdevice (see include/linux/netdevice.h)
4. What use_carrier options for link status detection does
OFED ipoib
support,
MII, ETHTOOL or netif_carrier_ok?
the mii/ethertool etc local link detection methods of the
bonding driver
are somehow deprecated, since nowadays almost any network device
support the netif_carrier_ok call. The --default-- of the upstream
bonding driver (eg the one we use in OFED and the 2.6.21
listed below)
is to set the use_carrier mod param to 1 that is mii is not
used anymore.
author: Thomas Davis, [EMAIL PROTECTED] and many others
description: Ethernet Channel Bonding Driver, v3.1.2
version: 3.1.2
parm: use_carrier:Use netif_carrier_ok (vs MII
ioctls) in miimon; 0 for off, 1 for on (default) (int)
parm: miimon:Link check interval in milliseconds (int)
If you have any good examples of bonding configuration
settings that work
with OFED, I'd appreciate that also.
The bonding RPM provided with OFED is made of a driver,
script and some
help text containing usage examples, please take a look there
and let me
know if you have further questions.
$ rpm -ql ib-bonding-0.9.0-2.6.9_42.ELsmp
/lib/modules/2.6.9-42.ELsmp/updates/kernel/drivers/net/bonding
/bonding.ko
/usr/bin/ib-bond
/usr/share/doc/ib-bonding-0.9.0/ib-bonding.txt
The ofed service (/etc/init.d/openibd) was enhanced to allow for
--persistent-- bonding configuration, please see the bonding
section at
docs/ipoib_release_notes.txt to see how to do it.
Or.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
