On 11/12/12 07:42, Mark Wu wrote:
> On 12/10/2012 08:24 PM, Antoni Segura Puimedon wrote:
>> Hello everybody,
>>
>> We found some unexpected behavior with bonds and we'd like to discuss it.
>> Please, read the forwarded messages.
>>
>> Best,
>>
>> Toni
>>
>> ----- Forwarded Message -----
>>> From: "Dan Kenigsberg" <dan...@redhat.com>
>>> To: "Antoni Segura Puimedon" <asegu...@redhat.com>
>>> Cc: "Livnat Peer" <lp...@redhat.com>, "Igor Lvovsky"
>>> <ilvov...@redhat.com>
>>> Sent: Monday, December 10, 2012 1:03:48 PM
>>> Subject: Re: Bonding, ifcfg and luck
>>>
>>> On Mon, Dec 10, 2012 at 06:47:58AM -0500, Antoni Segura Puimedon
>>> wrote:
>>>> Hi all,
>>>>
>>>> I discussed this briefly with Livnat over the phone and mentioned
>>>> it to Dan.
>>>> The issue that we have is that, if I understand correctly our
>>>> current
>>>> configNetwork, it could very well be that it works by means of good
>>>> design with
>>>> a side-dish of luck.
>>>>
>>>> I'll explain myself:
>>>> By design, as documented in
>>>> http://www.kernel.org/doc/Documentation/networking/bonding.txt:
>>>> "All slaves of bond0 have the same MAC address (HWaddr) as bond0
>>>> for all modes
>>>> except TLB and ALB that require a unique MAC address for each
>>>> slave."
>>>>
>>>> Thus, all operations on the slave interfaces after they are added
>>>> to the bond
>>>> (except on TLB and ALB modes) that rely on ifcfg will fail with a
>>>> message like:
>>>> "Device eth3 has different MAC address than expected, ignoring.",
>>>> and no
>>>> ifup/ifdown will be performed.
>>>>
>>>> Currently, we were not noticing this, because we were ignoring
>>>> completely
>>>> errors in ifdown and ifup, but http://gerrit.ovirt.org/#/c/8415/
>>>> shed light on
>>>> the matter. As you can see in the following example (bonding mode
>>>> 4) the
>>>> behavior is just as documented:
>>>>
>>>>      [root@rhel64 ~]# cat /sys/class/net/eth*/address
>>>>      52:54:00:a2:b4:50
>>>>      52:54:00:3f:9b:28
>>>>      52:54:00:51:50:49
>>>>      52:54:00:ac:32:1b <-----------------
>>>>      [root@rhel64 ~]# echo "+eth2" >
>>>>      /sys/class/net/bond0/bonding/slaves
>>>>      [root@rhel64 ~]# echo "+eth3" >
>>>>      /sys/class/net/bond0/bonding/slaves
>>>>      [root@rhel64 ~]# cat /sys/class/net/eth*/address
>>>>      52:54:00:a2:b4:50
>>>>      52:54:00:3f:9b:28
>>>>      52:54:00:51:50:49
>>>>      52:54:00:51:50:49 <-----------------
>>>>      [root@rhel64 ~]# echo "-eth3" >
>>>>      /sys/class/net/bond0/bonding/slaves
>>>>      [root@rhel64 ~]# cat /sys/class/net/eth*/address
>>>>      52:54:00:a2:b4:50
>>>>      52:54:00:3f:9b:28
>>>>      52:54:00:51:50:49
>>>>      52:54:00:ac:32:1b <-----------------
>>>>
>>>> Obviously, this means that, for example, when we add a bridge on
>>>> top of a bond,
>>>> the ifdown, ifup of the bond slaves will be completely fruitless
>>>> (although
>>>> luckily that doesn't prevent them from working).
>>>
>>> Sorry, thi is not obvious to me.
>>> When we change something in a nic, we first take it down (which break
>>> it
>>> away from the bond), change it, and then take it up again (and back
>>> to
>>> the bond).
>>>
>>> I did not understand which flow of configuration leads us to the
>>> "unexpected mac" error. I hope that we can circumvent it.
> I get the same question.  The warning message should be only seen when
> you run ifup on bonding or its slave, which is already up, otherwise
> the slave nic's mac address should hold its own permanent mac address.  
> If the bonding is down before, you shouldn't see this message
> because the nic is not enslaved.
>>>
>>>
>>>> To solve this issue on the ifcfg based operation we could either:
>>>> - Continue ignoring these issues and either not do ifup ifdown for
>>>> bonding
>>>>    slaves or catch the specific error and ignore it.
>>> That's reasonable, for a hack.
>>>
>>>> - Modify the ifcfg files of the slaves after they are enslaved to
>>>> reflect the
>>>>    MAC addr of /sys/class/net/bond0/address. Modify the ifcfg files
>>>>    after the
>>>>    bond is destroyed to reflect their own addresses as in
>>>>    /sys/class/net/ethx/address
>>> I do not undestand this solution at all... Fixing initscripts to
>>> expect
>>> the permanent mac address instead of the bond's one makes more sense
>>> to
>>> me. ( /proc/net/bonding/bond0 has "Permanent HW addr: " )
>>>
>>>> Livnat made me note that this behavior can be a problem to the anti
>>>> mac-spoofing rules that we add to iptables, as they rely on the
>>>> identity device
>>>> -macaddr to work and, obviously, in most bonding modes that is
>>>> broken unless
>>>> the device's macaddr is the one chosen for the bond.
>>> Right. I suppose we can open a bug about it: in-guest bond does not
>>> work
>>> with mac-no-spoofing. I have a vague memory of discussing this with
>>> lpeer few months back, but it somehow slipped my mind.
> I am not sure why we need bonding inside guest.  Bonding can provide
> link failover and bandwidth aggregation.
> We can achieve that by setting up bonding on host.  If we really need
> it,  we could workaround it by defining a netfilter for each vm,
> which allow the traffic from all the mac addresses belong to it.  To be
> exact,  we also could make qemu generate an event
> to libvirt on guest vif's mac address change, and then libvirt can
> validate the change and updating it's data accordingly.
> 

Network configuration in the guest is not something vdsm does (at least
not today) but our users can configure their guests anyway they want/need.
I think it is worth noting that if they configure a bond in the guest
and the default behavior today is to add mac-anti-spoofing rules for all
vNICs then the traffic will be blocked.

I agree that this is not a very interesting use case thus I would not go
after fixing it at this point.

>>>
>>>
>>>> Well, I think that is all for this issue. We should discuss which
>>>> is the best
>>>> approach for this before we move on with patches that account for
>>>> ifup ifdown
>>>> return information.
>>>>
>>>> Best,
>>>>
>>>> Toni
>>>>
>> _______________________________________________
>> vdsm-devel mailing list
>> vdsm-devel@lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel@lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

_______________________________________________
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

Reply via email to