You should use openibd.conf, not ifcfg-*, for configuring bonding at boot time.
Scott > -----Original Message----- > From: Bob Kossey [mailto:[EMAIL PROTECTED] > Sent: Wednesday, June 06, 2007 11:53 AM > To: Scott Weitzenkamp (sweitzen) > Cc: Or Gerlitz; OpenFabrics General > Subject: Re: [ofa-general] Re: ipoib / bonding and OFED > > Just to follow up on this, using RHEL5 and OFED 1.2 rc4, I was able > to do enough rudimentary testing to convince myself that IB > bonding was working. I was able to use ib-bond, as well > as the use of the openib.conf file to enable bonding on startup, > including both(separately) IPOIBBOND_ENABLE and IPOIBHA_ENABLE. > > One thing I was not able to do however, was to start IB bonding > using the standard bonding modifications to /etc/modprobe.conf > and /etc/sysconfig/network-scripts/ifcfg* files. Should this > be possible, > and are there perhaps some required settings I am missing? I'll > include my file modifications and some output below. > > modprobe.conf: > alias bond0 bonding > options bond0 mode=active-backup miimon=100 > > ifcfg-bond0: > DEVICE=bond0 > IPADDR="172.22.0.23" > NETMASK="255.255.0.0" > NETWORK="172.22.0.0" > BROADCAST="172.22.255.255" > ONBOOT=yes > BOOTPROTO=none > USERCTL=no > BONDING_SLAVE0=ib0 > BONDING_SLAVE0=ib1 > > ifcfg-ib0: > DEVICE=ib0 > USERCTL=no > ONBOOT=yes > MASTER=bond0 > SLAVE=yes > BOOTPROTO=none > > ifcfg-ib1: > DEVICE=ib1 > USERCTL=no > ONBOOT=yes > MASTER=bond0 > SLAVE=yes > BOOTPROTO=none > > [EMAIL PROTECTED] ~]# ifconfig > bond0 Link encap:InfiniBand HWaddr > 80:00:04:05:FE:80:00:00:00:00:00:00:00:0 > 0:00:00:00:00:00:00 > inet addr:172.22.0.23 Bcast:172.22.255.255 Mask:255.255.0.0 > UP BROADCAST MASTER MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:18 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 b) TX bytes:1352 (1.3 KiB) > > dmesg: > ... > Ethernet Channel Bonding Driver: v3.1.1 (September 26, 2006) > bonding: MII link monitoring set to 100 ms > ADDRCONF(NETDEV_UP): bond0: link is not ready > bonding: bond0: Adding slave ib0. > bonding: bond0: Warning: enslaved VLAN challenged slave ib0. Adding > VLANs will b > e blocked as long as ib0 is part of bond bond0 > bonding: bond0: Warning: The first slave device you specified > does not > support s > etting the MAC address. This bond MAC address would be that of the > active slave. > ADDRCONF(NETDEV_UP): ib0: link is not ready > bonding: bond0: Warning: failed to get speed and duplex from ib0, > assumed to be > 100Mb/sec and Full. > bonding: bond0: making interface ib0 the new active one. > bondingbond_send_grat_arp: bond bond0 slave ib0 > bonding: bond0: first active interface up! > bonding: bond0: enslaving ib0 as an active interface with an up link. > bonding: bond0: Adding slave ib1. > bonding: bond0: Warning: enslaved VLAN challenged slave ib1. Adding > VLANs will b > e blocked as long as ib1 is part of bond bond0 > ADDRCONF(NETDEV_UP): ib1: link is not ready > bonding: bond0: Warning: failed to get speed and duplex from ib1, > assumed to be > 100Mb/sec and Full. > bonding: bond0: enslaving ib1 as a backup interface with an up link. > ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready > ... > bonding: bond0: Interface ib0 is already enslaved! > ib0: enabling connected mode will cause multicast packet drops > ib0: mtu > 2044 will cause multicast packet drops. > bonding: bond0: link status definitely down for interface > ib0, disabling it > bonding: bond0: making interface ib1 the new active one. > bondingbond_send_grat_arp: bond bond0 slave ib1 > bonding: bond0: Interface ib1 is already enslaved! > ib1: enabling connected mode will cause multicast packet drops > ib1: mtu > 2044 will cause multicast packet drops. > bonding: bond0: link status definitely down for interface > ib1, disabling it > bondingbond_send_grat_arp: bond bond0 slave NULL > bonding: bond0: now running without any active interface ! > > Thanks, > Bob > > Scott Weitzenkamp (sweitzen) wrote: > > Bob, it is now possible to configure IPoIB bonding in > > /etc/infiniband/openib.conf, this configuration file includes the > > following boilerplate. > > > > # Enable the bonding driver on startup > > IPOIBBOND_ENABLE=no > > # Set bond interface names > > #IPOIB_BONDS=bond0,bond1 > > # Set specific bond params; address and slaves > > #bond0_IP=10.10.10.1 > > #bond0_SLAVES=ib0,ib1 > > #bond1_IP=20.10.10.1 > > #bond1_SLAVES=ib2,ib3,ib4 > > > > Scott Weitzenkamp > > SQA and Release Manager > > Server Virtualization Business Unit > > Cisco Systems > > > > > > > >> -----Original Message----- > >> From: [EMAIL PROTECTED] > >> [mailto:[EMAIL PROTECTED] On Behalf > Of Or Gerlitz > >> Sent: Tuesday, May 29, 2007 12:56 AM > >> To: Bob Kossey > >> Cc: OpenFabrics General > >> Subject: [ofa-general] Re: ipoib / bonding and OFED > >> > >> Bob Kossey wrote: > >> > >>> I copied OR since I think this is related to his OFED HA work, and > >>> he might have some insights. A few more questions for Or: > >>> I was trying to use ipoib bonding with OFED 1.2 rc2 and a > >>> > >> 2.6.9 kernel, > >> > >>> but was not able to get it to work so far. I saw your > >>> > >> Sonoma bonding > >> > >>> slides, and you mention kernel bonding driver changes were needed. > >>> 2. Is there a minimum kernel version, with the kernel > bonding driver > >>> changes, that is required to use bonding with OFED ipoib? > >>> > >> Just to have a base line here: to get bonding to work with > IPoIB, you > >> should use the bonding driver provided with OFED 1.2. This > >> driver is the > >> upstream one (of 2.6.20) being patched to support IPoIB and > >> backported > >> to RH5, SLES10 and RH4 U3/4/5, other kernels are not supported. > >> > >> If you were using the ofed bonding on a system that matches > >> the support > >> matrix it should worl. If do have problems under this > config, please > >> either open a bug at the ofed bugzilla > >> @ bugs.openfabrics.org assigned to [EMAIL PROTECTED] > (Moni Shoua) or > >> send first report/question to Moni and CC [EMAIL PROTECTED] > >> > >> Please note that between RC2 and RC4 (to be released today > etc) some > >> bugs were fixed, you can search in the bugzilla to see what. > >> > >> > >>> 3. The bonding driver uses the HWADDR from the underlying ipoib > >>> devices, how does it obtain the HWADDR? Does it use the > >>> > >> full 20 bytes, > >> > >>> or some subset? > >>> > >> when enslaving IPoIB devices, the bonding driver uses the full hw > >> address of the active slave, it simply looks on the > dev_addr field of > >> the slave struct netdevice (see include/linux/netdevice.h) > >> > >> > >>> 4. What use_carrier options for link status detection does > >>> > >> OFED ipoib > >> > >>> support, > >>> MII, ETHTOOL or netif_carrier_ok? > >>> > >> the mii/ethertool etc local link detection methods of the > >> bonding driver > >> are somehow deprecated, since nowadays almost any network device > >> support the netif_carrier_ok call. The --default-- of the upstream > >> bonding driver (eg the one we use in OFED and the 2.6.21 > >> listed below) > >> is to set the use_carrier mod param to 1 that is mii is not > >> used anymore. > >> > >> > >>> author: Thomas Davis, [EMAIL PROTECTED] and many others > >>> description: Ethernet Channel Bonding Driver, v3.1.2 > >>> version: 3.1.2 > >>> parm: use_carrier:Use netif_carrier_ok (vs MII > >>> > >> ioctls) in miimon; 0 for off, 1 for on (default) (int) > >> > >>> parm: miimon:Link check interval in milliseconds (int) > >>> > >>> If you have any good examples of bonding configuration > >>> > >> settings that work > >> > >>> with OFED, I'd appreciate that also. > >>> > >> The bonding RPM provided with OFED is made of a driver, > >> script and some > >> help text containing usage examples, please take a look there > >> and let me > >> know if you have further questions. > >> > >> > >>> $ rpm -ql ib-bonding-0.9.0-2.6.9_42.ELsmp > >>> > >>> > >> /lib/modules/2.6.9-42.ELsmp/updates/kernel/drivers/net/bonding > >> /bonding.ko > >> > >>> /usr/bin/ib-bond > >>> /usr/share/doc/ib-bonding-0.9.0/ib-bonding.txt > >>> > >> The ofed service (/etc/init.d/openibd) was enhanced to allow for > >> --persistent-- bonding configuration, please see the bonding > >> section at > >> docs/ipoib_release_notes.txt to see how to do it. > >> > >> Or. > >> > >> _______________________________________________ > >> general mailing list > >> [email protected] > >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >> > >> To unsubscribe, please visit > >> http://openib.org/mailman/listinfo/openib-general > >> > >> > _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
