On 11/12/12 07:42, Mark Wu wrote:
> On 12/10/2012 08:24 PM, Antoni Segura Puimedon wrote:
>> Hello everybody,
>> We found some unexpected behavior with bonds and we'd like to discuss it.
>> Please, read the forwarded messages.
>> Best,
>> Toni
>> ----- Forwarded Message -----
>>> From: "Dan Kenigsberg" <dan...@redhat.com>
>>> To: "Antoni Segura Puimedon" <asegu...@redhat.com>
>>> Cc: "Livnat Peer" <lp...@redhat.com>, "Igor Lvovsky"
>>> <ilvov...@redhat.com>
>>> Sent: Monday, December 10, 2012 1:03:48 PM
>>> Subject: Re: Bonding, ifcfg and luck
>>> On Mon, Dec 10, 2012 at 06:47:58AM -0500, Antoni Segura Puimedon
>>> wrote:
>>>> Hi all,
>>>> I discussed this briefly with Livnat over the phone and mentioned
>>>> it to Dan.
>>>> The issue that we have is that, if I understand correctly our
>>>> current
>>>> configNetwork, it could very well be that it works by means of good
>>>> design with
>>>> a side-dish of luck.
>>>> I'll explain myself:
>>>> By design, as documented in
>>>> http://www.kernel.org/doc/Documentation/networking/bonding.txt:
>>>> "All slaves of bond0 have the same MAC address (HWaddr) as bond0
>>>> for all modes
>>>> except TLB and ALB that require a unique MAC address for each
>>>> slave."
>>>> Thus, all operations on the slave interfaces after they are added
>>>> to the bond
>>>> (except on TLB and ALB modes) that rely on ifcfg will fail with a
>>>> message like:
>>>> "Device eth3 has different MAC address than expected, ignoring.",
>>>> and no
>>>> ifup/ifdown will be performed.
>>>> Currently, we were not noticing this, because we were ignoring
>>>> completely
>>>> errors in ifdown and ifup, but http://gerrit.ovirt.org/#/c/8415/
>>>> shed light on
>>>> the matter. As you can see in the following example (bonding mode
>>>> 4) the
>>>> behavior is just as documented:
>>>>      [root@rhel64 ~]# cat /sys/class/net/eth*/address
>>>>      52:54:00:a2:b4:50
>>>>      52:54:00:3f:9b:28
>>>>      52:54:00:51:50:49
>>>>      52:54:00:ac:32:1b <-----------------
>>>>      [root@rhel64 ~]# echo "+eth2" >
>>>>      /sys/class/net/bond0/bonding/slaves
>>>>      [root@rhel64 ~]# echo "+eth3" >
>>>>      /sys/class/net/bond0/bonding/slaves
>>>>      [root@rhel64 ~]# cat /sys/class/net/eth*/address
>>>>      52:54:00:a2:b4:50
>>>>      52:54:00:3f:9b:28
>>>>      52:54:00:51:50:49
>>>>      52:54:00:51:50:49 <-----------------
>>>>      [root@rhel64 ~]# echo "-eth3" >
>>>>      /sys/class/net/bond0/bonding/slaves
>>>>      [root@rhel64 ~]# cat /sys/class/net/eth*/address
>>>>      52:54:00:a2:b4:50
>>>>      52:54:00:3f:9b:28
>>>>      52:54:00:51:50:49
>>>>      52:54:00:ac:32:1b <-----------------
>>>> Obviously, this means that, for example, when we add a bridge on
>>>> top of a bond,
>>>> the ifdown, ifup of the bond slaves will be completely fruitless
>>>> (although
>>>> luckily that doesn't prevent them from working).
>>> Sorry, thi is not obvious to me.
>>> When we change something in a nic, we first take it down (which break
>>> it
>>> away from the bond), change it, and then take it up again (and back
>>> to
>>> the bond).
>>> I did not understand which flow of configuration leads us to the
>>> "unexpected mac" error. I hope that we can circumvent it.
> I get the same question.  The warning message should be only seen when
> you run ifup on bonding or its slave, which is already up, otherwise
> the slave nic's mac address should hold its own permanent mac address.  
> If the bonding is down before, you shouldn't see this message
> because the nic is not enslaved.
>>>> To solve this issue on the ifcfg based operation we could either:
>>>> - Continue ignoring these issues and either not do ifup ifdown for
>>>> bonding
>>>>    slaves or catch the specific error and ignore it.
>>> That's reasonable, for a hack.
>>>> - Modify the ifcfg files of the slaves after they are enslaved to
>>>> reflect the
>>>>    MAC addr of /sys/class/net/bond0/address. Modify the ifcfg files
>>>>    after the
>>>>    bond is destroyed to reflect their own addresses as in
>>>>    /sys/class/net/ethx/address
>>> I do not undestand this solution at all... Fixing initscripts to
>>> expect
>>> the permanent mac address instead of the bond's one makes more sense
>>> to
>>> me. ( /proc/net/bonding/bond0 has "Permanent HW addr: " )
>>>> Livnat made me note that this behavior can be a problem to the anti
>>>> mac-spoofing rules that we add to iptables, as they rely on the
>>>> identity device
>>>> -macaddr to work and, obviously, in most bonding modes that is
>>>> broken unless
>>>> the device's macaddr is the one chosen for the bond.
>>> Right. I suppose we can open a bug about it: in-guest bond does not
>>> work
>>> with mac-no-spoofing. I have a vague memory of discussing this with
>>> lpeer few months back, but it somehow slipped my mind.
> I am not sure why we need bonding inside guest.  Bonding can provide
> link failover and bandwidth aggregation.
> We can achieve that by setting up bonding on host.  If we really need
> it,  we could workaround it by defining a netfilter for each vm,
> which allow the traffic from all the mac addresses belong to it.  To be
> exact,  we also could make qemu generate an event
> to libvirt on guest vif's mac address change, and then libvirt can
> validate the change and updating it's data accordingly.

Network configuration in the guest is not something vdsm does (at least
not today) but our users can configure their guests anyway they want/need.
I think it is worth noting that if they configure a bond in the guest
and the default behavior today is to add mac-anti-spoofing rules for all
vNICs then the traffic will be blocked.

I agree that this is not a very interesting use case thus I would not go
after fixing it at this point.

>>>> Well, I think that is all for this issue. We should discuss which
>>>> is the best
>>>> approach for this before we move on with patches that account for
>>>> ifup ifdown
>>>> return information.
>>>> Best,
>>>> Toni
>> _______________________________________________
>> vdsm-devel mailing list
>> vdsm-devel@lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel@lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

vdsm-devel mailing list

Reply via email to