----- Original Message -----
> From: "Mark Wu" <wu...@linux.vnet.ibm.com>
> To: "Antoni Segura Puimedon" <asegu...@redhat.com>
> Cc: vdsm-devel@lists.fedorahosted.org
> Sent: Tuesday, December 11, 2012 6:42:45 AM
> Subject: Re: [vdsm] Fwd: Bonding, bridges and ifcfg
> 
> On 12/10/2012 08:24 PM, Antoni Segura Puimedon wrote:
> > Hello everybody,
> >
> > We found some unexpected behavior with bonds and we'd like to
> > discuss it.
> > Please, read the forwarded messages.
> >
> > Best,
> >
> > Toni
> >
> > ----- Forwarded Message -----
> >> From: "Dan Kenigsberg" <dan...@redhat.com>
> >> To: "Antoni Segura Puimedon" <asegu...@redhat.com>
> >> Cc: "Livnat Peer" <lp...@redhat.com>, "Igor Lvovsky"
> >> <ilvov...@redhat.com>
> >> Sent: Monday, December 10, 2012 1:03:48 PM
> >> Subject: Re: Bonding, ifcfg and luck
> >>
> >> On Mon, Dec 10, 2012 at 06:47:58AM -0500, Antoni Segura Puimedon
> >> wrote:
> >>> Hi all,
> >>>
> >>> I discussed this briefly with Livnat over the phone and mentioned
> >>> it to Dan.
> >>> The issue that we have is that, if I understand correctly our
> >>> current
> >>> configNetwork, it could very well be that it works by means of
> >>> good
> >>> design with
> >>> a side-dish of luck.
> >>>
> >>> I'll explain myself:
> >>> By design, as documented in
> >>> http://www.kernel.org/doc/Documentation/networking/bonding.txt:
> >>> "All slaves of bond0 have the same MAC address (HWaddr) as bond0
> >>> for all modes
> >>> except TLB and ALB that require a unique MAC address for each
> >>> slave."
> >>>
> >>> Thus, all operations on the slave interfaces after they are added
> >>> to the bond
> >>> (except on TLB and ALB modes) that rely on ifcfg will fail with a
> >>> message like:
> >>> "Device eth3 has different MAC address than expected, ignoring.",
> >>> and no
> >>> ifup/ifdown will be performed.
> >>>
> >>> Currently, we were not noticing this, because we were ignoring
> >>> completely
> >>> errors in ifdown and ifup, but http://gerrit.ovirt.org/#/c/8415/
> >>> shed light on
> >>> the matter. As you can see in the following example (bonding mode
> >>> 4) the
> >>> behavior is just as documented:
> >>>
> >>>      [root@rhel64 ~]# cat /sys/class/net/eth*/address
> >>>      52:54:00:a2:b4:50
> >>>      52:54:00:3f:9b:28
> >>>      52:54:00:51:50:49
> >>>      52:54:00:ac:32:1b <-----------------
> >>>      [root@rhel64 ~]# echo "+eth2" >
> >>>      /sys/class/net/bond0/bonding/slaves
> >>>      [root@rhel64 ~]# echo "+eth3" >
> >>>      /sys/class/net/bond0/bonding/slaves
> >>>      [root@rhel64 ~]# cat /sys/class/net/eth*/address
> >>>      52:54:00:a2:b4:50
> >>>      52:54:00:3f:9b:28
> >>>      52:54:00:51:50:49
> >>>      52:54:00:51:50:49 <-----------------
> >>>      [root@rhel64 ~]# echo "-eth3" >
> >>>      /sys/class/net/bond0/bonding/slaves
> >>>      [root@rhel64 ~]# cat /sys/class/net/eth*/address
> >>>      52:54:00:a2:b4:50
> >>>      52:54:00:3f:9b:28
> >>>      52:54:00:51:50:49
> >>>      52:54:00:ac:32:1b <-----------------
> >>>
> >>> Obviously, this means that, for example, when we add a bridge on
> >>> top of a bond,
> >>> the ifdown, ifup of the bond slaves will be completely fruitless
> >>> (although
> >>> luckily that doesn't prevent them from working).
> >>
> >> Sorry, thi is not obvious to me.
> >> When we change something in a nic, we first take it down (which
> >> break
> >> it
> >> away from the bond), change it, and then take it up again (and
> >> back
> >> to
> >> the bond).
> >>
> >> I did not understand which flow of configuration leads us to the
> >> "unexpected mac" error. I hope that we can circumvent it.
> I get the same question.  The warning message should be only seen
> when
> you run ifup on bonding or its slave, which is already up, otherwise
> the slave nic's mac address should hold its own permanent mac
> address.
> If the bonding is down before, you shouldn't see this message
> because the nic is not enslaved.

It's easy enough to reproduce. To the current HEAD do:

vdsClient 0 addNetwork bridge=kaboom bonding=bond0 nics=eth1,eth2 bridged=True

Go to /var/log/vdsm/vdsm.log and you'll see that:
http://pastebin.com/09eRNzv6

On line 33 you'll see that the ifup of eth2 is FAILED.

> >>
> >>
> >>> To solve this issue on the ifcfg based operation we could either:
> >>> - Continue ignoring these issues and either not do ifup ifdown
> >>> for
> >>> bonding
> >>>    slaves or catch the specific error and ignore it.
> >> That's reasonable, for a hack.
> >>
> >>> - Modify the ifcfg files of the slaves after they are enslaved to
> >>> reflect the
> >>>    MAC addr of /sys/class/net/bond0/address. Modify the ifcfg
> >>>    files
> >>>    after the
> >>>    bond is destroyed to reflect their own addresses as in
> >>>    /sys/class/net/ethx/address
> >> I do not undestand this solution at all... Fixing initscripts to
> >> expect
> >> the permanent mac address instead of the bond's one makes more
> >> sense
> >> to
> >> me. ( /proc/net/bonding/bond0 has "Permanent HW addr: " )
> >>
> >>> Livnat made me note that this behavior can be a problem to the
> >>> anti
> >>> mac-spoofing rules that we add to iptables, as they rely on the
> >>> identity device
> >>> -macaddr to work and, obviously, in most bonding modes that is
> >>> broken unless
> >>> the device's macaddr is the one chosen for the bond.
> >> Right. I suppose we can open a bug about it: in-guest bond does
> >> not
> >> work
> >> with mac-no-spoofing. I have a vague memory of discussing this
> >> with
> >> lpeer few months back, but it somehow slipped my mind.
> I am not sure why we need bonding inside guest.  Bonding can provide
> link failover and bandwidth aggregation.
> We can achieve that by setting up bonding on host.  If we really need
> it,  we could workaround it by defining a netfilter for each vm,
> which allow the traffic from all the mac addresses belong to it.  To
> be
> exact,  we also could make qemu generate an event
> to libvirt on guest vif's mac address change, and then libvirt can
> validate the change and updating it's data accordingly.
> 
> >>
> >>
> >>> Well, I think that is all for this issue. We should discuss which
> >>> is the best
> >>> approach for this before we move on with patches that account for
> >>> ifup ifdown
> >>> return information.
> >>>
> >>> Best,
> >>>
> >>> Toni
> >>>
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel@lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
> 
_______________________________________________
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

Reply via email to