----- Original Message ----- > From: "Mark Wu" <wu...@linux.vnet.ibm.com> > To: "Igor Lvovsky" <ilvov...@redhat.com> > Cc: "Antoni Segura Puimedon" <asegu...@redhat.com>, > vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" > <dan...@redhat.com> > Sent: Wednesday, December 12, 2012 8:22:29 AM > Subject: Re: [vdsm] Fwd: Bonding, bridges and ifcfg > > On 12/12/2012 03:13 PM, Igor Lvovsky wrote: > > > > ----- Original Message ----- > >> From: "Antoni Segura Puimedon" <asegu...@redhat.com> > >> To: "Dan Kenigsberg" <dan...@redhat.com> > >> Cc: "Mark Wu" <wu...@linux.vnet.ibm.com>, > >> vdsm-devel@lists.fedorahosted.org, "Igor Lvovsky" > >> <ilvov...@redhat.com> > >> Sent: Wednesday, December 12, 2012 1:45:54 AM > >> Subject: Re: [vdsm] Fwd: Bonding, bridges and ifcfg > >> > >> > >> > >> ----- Original Message ----- > >>> From: "Dan Kenigsberg" <dan...@redhat.com> > >>> To: "Antoni Segura Puimedon" <asegu...@redhat.com> > >>> Cc: "Mark Wu" <wu...@linux.vnet.ibm.com>, > >>> vdsm-devel@lists.fedorahosted.org, "Igor Lvovsky" > >>> <ilvov...@redhat.com> > >>> Sent: Tuesday, December 11, 2012 10:38:45 PM > >>> Subject: Re: [vdsm] Fwd: Bonding, bridges and ifcfg > >>> > >>> On Tue, Dec 11, 2012 at 04:21:51PM -0500, Antoni Segura Puimedon > >>> wrote: > >>>> > >>>> ----- Original Message ----- > >>>>> From: "Dan Kenigsberg" <dan...@redhat.com> > >>>>> To: "Antoni Segura Puimedon" <asegu...@redhat.com>, "Igor > >>>>> Lvovsky" <ilvov...@redhat.com> > >>>>> Cc: "Mark Wu" <wu...@linux.vnet.ibm.com>, > >>>>> vdsm-devel@lists.fedorahosted.org > >>>>> Sent: Tuesday, December 11, 2012 7:44:05 PM > >>>>> Subject: Re: [vdsm] Fwd: Bonding, bridges and ifcfg > >>>>> > >>>>> On Tue, Dec 11, 2012 at 11:42:49AM -0500, Antoni Segura > >>>>> Puimedon > >>>>> wrote: > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>>> From: "Mark Wu" <wu...@linux.vnet.ibm.com> > >>>>>>> To: "Antoni Segura Puimedon" <asegu...@redhat.com> > >>>>>>> Cc: vdsm-devel@lists.fedorahosted.org > >>>>>>> Sent: Tuesday, December 11, 2012 6:42:45 AM > >>>>>>> Subject: Re: [vdsm] Fwd: Bonding, bridges and ifcfg > >>>>>>> > >>>>>>> On 12/10/2012 08:24 PM, Antoni Segura Puimedon wrote: > >>>>>>>> Hello everybody, > >>>>>>>> > >>>>>>>> We found some unexpected behavior with bonds and we'd > >>>>>>>> like > >>>>>>>> to > >>>>>>>> discuss it. > >>>>>>>> Please, read the forwarded messages. > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> > >>>>>>>> Toni > >>>>>>>> > >>>>>>>> ----- Forwarded Message ----- > >>>>>>>>> From: "Dan Kenigsberg" <dan...@redhat.com> > >>>>>>>>> To: "Antoni Segura Puimedon" <asegu...@redhat.com> > >>>>>>>>> Cc: "Livnat Peer" <lp...@redhat.com>, "Igor Lvovsky" > >>>>>>>>> <ilvov...@redhat.com> > >>>>>>>>> Sent: Monday, December 10, 2012 1:03:48 PM > >>>>>>>>> Subject: Re: Bonding, ifcfg and luck > >>>>>>>>> > >>>>>>>>> On Mon, Dec 10, 2012 at 06:47:58AM -0500, Antoni Segura > >>>>>>>>> Puimedon > >>>>>>>>> wrote: > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> I discussed this briefly with Livnat over the phone and > >>>>>>>>>> mentioned > >>>>>>>>>> it to Dan. > >>>>>>>>>> The issue that we have is that, if I understand > >>>>>>>>>> correctly > >>>>>>>>>> our > >>>>>>>>>> current > >>>>>>>>>> configNetwork, it could very well be that it works by > >>>>>>>>>> means > >>>>>>>>>> of > >>>>>>>>>> good > >>>>>>>>>> design with > >>>>>>>>>> a side-dish of luck. > >>>>>>>>>> > >>>>>>>>>> I'll explain myself: > >>>>>>>>>> By design, as documented in > >>>>>>>>>> http://www.kernel.org/doc/Documentation/networking/bonding.txt: > >>>>>>>>>> "All slaves of bond0 have the same MAC address (HWaddr) > >>>>>>>>>> as > >>>>>>>>>> bond0 > >>>>>>>>>> for all modes > >>>>>>>>>> except TLB and ALB that require a unique MAC address > >>>>>>>>>> for > >>>>>>>>>> each > >>>>>>>>>> slave." > >>>>>>>>>> > >>>>>>>>>> Thus, all operations on the slave interfaces after they > >>>>>>>>>> are > >>>>>>>>>> added > >>>>>>>>>> to the bond > >>>>>>>>>> (except on TLB and ALB modes) that rely on ifcfg will > >>>>>>>>>> fail > >>>>>>>>>> with a > >>>>>>>>>> message like: > >>>>>>>>>> "Device eth3 has different MAC address than expected, > >>>>>>>>>> ignoring.", > >>>>>>>>>> and no > >>>>>>>>>> ifup/ifdown will be performed. > >>>>>>>>>> > >>>>>>>>>> Currently, we were not noticing this, because we were > >>>>>>>>>> ignoring > >>>>>>>>>> completely > >>>>>>>>>> errors in ifdown and ifup, but > >>>>>>>>>> http://gerrit.ovirt.org/#/c/8415/ > >>>>>>>>>> shed light on > >>>>>>>>>> the matter. As you can see in the following example > >>>>>>>>>> (bonding > >>>>>>>>>> mode > >>>>>>>>>> 4) the > >>>>>>>>>> behavior is just as documented: > >>>>>>>>>> > >>>>>>>>>> [root@rhel64 ~]# cat /sys/class/net/eth*/address > >>>>>>>>>> 52:54:00:a2:b4:50 > >>>>>>>>>> 52:54:00:3f:9b:28 > >>>>>>>>>> 52:54:00:51:50:49 > >>>>>>>>>> 52:54:00:ac:32:1b <----------------- > >>>>>>>>>> [root@rhel64 ~]# echo "+eth2" > > >>>>>>>>>> /sys/class/net/bond0/bonding/slaves > >>>>>>>>>> [root@rhel64 ~]# echo "+eth3" > > >>>>>>>>>> /sys/class/net/bond0/bonding/slaves > >>>>>>>>>> [root@rhel64 ~]# cat /sys/class/net/eth*/address > >>>>>>>>>> 52:54:00:a2:b4:50 > >>>>>>>>>> 52:54:00:3f:9b:28 > >>>>>>>>>> 52:54:00:51:50:49 > >>>>>>>>>> 52:54:00:51:50:49 <----------------- > >>>>>>>>>> [root@rhel64 ~]# echo "-eth3" > > >>>>>>>>>> /sys/class/net/bond0/bonding/slaves > >>>>>>>>>> [root@rhel64 ~]# cat /sys/class/net/eth*/address > >>>>>>>>>> 52:54:00:a2:b4:50 > >>>>>>>>>> 52:54:00:3f:9b:28 > >>>>>>>>>> 52:54:00:51:50:49 > >>>>>>>>>> 52:54:00:ac:32:1b <----------------- > >>>>>>>>>> > >>>>>>>>>> Obviously, this means that, for example, when we add a > >>>>>>>>>> bridge > >>>>>>>>>> on > >>>>>>>>>> top of a bond, > >>>>>>>>>> the ifdown, ifup of the bond slaves will be completely > >>>>>>>>>> fruitless > >>>>>>>>>> (although > >>>>>>>>>> luckily that doesn't prevent them from working). > >>>>>>>>> Sorry, thi is not obvious to me. > >>>>>>>>> When we change something in a nic, we first take it down > >>>>>>>>> (which > >>>>>>>>> break > >>>>>>>>> it > >>>>>>>>> away from the bond), change it, and then take it up > >>>>>>>>> again > >>>>>>>>> (and > >>>>>>>>> back > >>>>>>>>> to > >>>>>>>>> the bond). > >>>>>>>>> > >>>>>>>>> I did not understand which flow of configuration leads > >>>>>>>>> us > >>>>>>>>> to > >>>>>>>>> the > >>>>>>>>> "unexpected mac" error. I hope that we can circumvent > >>>>>>>>> it. > >>>>>>> I get the same question. The warning message should be > >>>>>>> only > >>>>>>> seen > >>>>>>> when > >>>>>>> you run ifup on bonding or its slave, which is already up, > >>>>>>> otherwise > >>>>>>> the slave nic's mac address should hold its own permanent > >>>>>>> mac > >>>>>>> address. > >>>>>>> If the bonding is down before, you shouldn't see this > >>>>>>> message > >>>>>>> because the nic is not enslaved. > >>>>>> It's easy enough to reproduce. To the current HEAD do: > >>>>>> > >>>>>> vdsClient 0 addNetwork bridge=kaboom bonding=bond0 > >>>>>> nics=eth1,eth2 > >>>>>> bridged=True > >>>>>> > >>>>>> Go to /var/log/vdsm/vdsm.log and you'll see that: > >>>>>> http://pastebin.com/09eRNzv6 > >>>>>> > >>>>>> On line 33 you'll see that the ifup of eth2 is FAILED. > >>>>> Ok, I think this is a bug in the sequence of our ifup'ing. > >>>>> Reviewing > >>>>> ifup-eth I see that in its "if bonding device" stanza there's > >>>>> > >>>>> if [ "$ISALIAS" = no ] && is_bonding_device ${DEVICE} ; then > >>>>> ... > >>>>> for device in $(LANG=C egrep -l > >>>>> "^[[:space:]]*MASTER=\"?${DEVICE}\"?[[:space:]]*$" > >>>>> /etc/sysconfig/network-scripts/ifcfg-*) ; do > >>>>> is_ignored_file "$device" && continue > >>>>> /sbin/ifup ${device##*/} > >>>>> ... > >>>>> > >>>>> meaning that if we prepare ifcfg-eth* files on time, the > >>>>> interfaces > >>>>> would be > >>>>> brought up by means of `ifup bond??`; and it seems that there > >>>>> is > >>>>> no > >>>>> need > >>>>> for a specific `ifup eth0` at all. > >>>> Yes this is the same conclusion I arrived when I looked at it > >>>> earlier, > >>>> Basically on bonding case we can just not do the ifups of the > >>>> underlying > >>>> devices. > >>> s/we can/we should/ . > >>> And it would also save us some wasted time. > >> Agreed! > > what about ifdown? Did you look at it? > > Is it redundant to ifdown NICs too if we ifdown bond? > Yes, the same as ifup: > > if is_bonding_device ${DEVICE} ; then > for device in $(LANG=C egrep -l > "^[[:space:]]*MASTER=\"?${DEVICE}\"?" > /etc/sysconfig/network-scripts/ifcfg-*) ; do > is_ignored_file "$device" && continue > /sbin/ifdown ${device##*/} > done > > > > >
I wonder if in F18 we are going to get bitten by NetworkManager with the mac address change. I will put on my TODO list to check that it doesn't regain control over the interface and if, submit a bug for it. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
