[ovs-discuss] ARP loop on OVS topology
Good afternoon, I hope someone could help me with the problem I'm going to describe. I have 3 virtual machines, each one with an OVS bridge "br0", which is isolated. The VM1 and VM2 are connected via a GRE tunnel to the Server and via VXLAN tunnels between both. The br0 of the Server (10.0.1.50) is the default gateway of the two VMS. https://i.stack.imgur.com/GDdqJ.jpg [https://i.stack.imgur.com/GDdqJ.jpg] Whenever I run: $ arping -I br0 10.0.1.10 (or) $ arping -I br0 10.0.1.20 from the server, an ARP loop takes place and the network become unusable, since all bridges become full with forwarded ARP requests and replies. In the network environment I'm trying to create, I don't want the VM1 br0 and the VM2 br0 to send ARPs to each other, in order to prevent this loop. So, in both VMs, I installed the following flow entries, once at a time, to see if any of them would solve the loop: $ ovs-ofctl -O openflow13 add-flow br0 priority=65535,arp,in_port=,arp_spa=10.0.1.0/24,action=drop $ ovs-ofctl -O openflow13 add-flow br0 priority=65535,arp,in_port=gre1,arp_spa=10.0.1.50,arp_tpa=10.0.1.20,action=drop (same for gre2 and 10.0.1.10) $ ovs-ofctl -O openflow13 add-flow br0 priority=65535,arp,in_port=,arp_tha=ff:ff:ff:ff:ff:ff,action=drop No success, the ARP loop persists and most of the time, these flow entries are ignored (no n_bytes counting). What is the correct way to formulate the flow entries so that ARP packets coming from the server, that are not destined to the bridge, are dropped? And how can I block ARP packets of the 10.0.1.0/24 network coming from the vxlan tunnel? In conclusion, how can I prevent this loop? Note: I've already enabled STP in the VM1 and VM2 bridges, but the problem persists. Thank you for your time and stay safe, Ines ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] No connectivity due to missing ARP reply
Thanks Michael for reporting this and Dumitru for fixing it!
@Michael, I guess that this bug is only relevant for the non-DVR case
right? ie. if the NAT entry for the FIP has both the external_mac and
logical_port fields, then the ARP reply will happen on the compute node
hosting the target VM and that'd be fine as it doesn't need to traverse the
gw node. Am I right?
On Tue, Mar 24, 2020 at 12:17 PM Dumitru Ceara wrote:
> On 3/24/20 8:50 AM, Plato, Michael wrote:
> > Hi Dumitru,
> >
> > thank you very much for the patch. I tried it and it works. VM1 can now
> reach VM2.
> >
> > Best regards!
> >
> > Michael
>
> Hi Michael,
>
> The fix is now merged in OVN master and branch 20.03:
>
>
> https://github.com/ovn-org/ovn/commit/d2ab98463f299e67a9f9a31e8b7c42680b8645cf
>
> Regards,
> Dumitru
>
> >
> >
> > -Ursprüngliche Nachricht-
> > Von: Dumitru Ceara
> > Gesendet: Montag, 23. März 2020 13:28
> > An: Plato, Michael ;
> ovs-discuss@openvswitch.org
> > Betreff: Re: [ovs-discuss] No connectivity due to missing ARP reply
> >
> > On 3/21/20 7:04 PM, Plato, Michael wrote:
> >>
> >> Hi all,
> >>
> >> we use OVN with Openstack and have a problem with the following setup:
> >>
> >>
> >> | |
> >> --- | 10.176.0.156 | ---
> >> | VM1 |- | 192.168.0.1|---| VM2 |
> >> --- | |
> ---
> >> 10.176.0.3.123 |--| R1 |-| 192.168.0.201 /
> GW: 192.168.0.1
> >> GW:10.176.0.1| |(test)| | FIP:
> 10.176.2.19
> >> | |
> >> Outside test
> >> (10.176.0.0/16) (192.168.0.0/24)
> >> (VLAN) (GENEVE)
> >>
> >>
> >> Versions:
> >> - OVN (20.03)
> >> - OVS (2.13)
> >> - networking-ovn (7.1.0)
> >>
> >> Problem:
> >> - no connectivity due to missing ARP reply for FIP 10.176.2.19 from
> >> VM1 (if VM1 is not on GW Chassis for R1 -> is_chassis_resident rules
> >> not applied)
> >> - after moving VM1 to chassis hosting R1 ARP reply appears (due to
> >> local "is_chassis_resident" ARP responder rules)
> >> - temporarily removing priority 75 rules (inserted by commit [0])
> >> restores functionality (even on non gateway chassis), because ARP
> >> requests were flooded to complete L2 domain (but this creates a
> >> scaling issue)
> >>
> >>
> >> Analysis:
> >> - according to ovs-detrace the ARP requests were dropped instead of
> >> being forwarded to remote chassis hosting R1 (as intended by [0])
> >>
> >>
> >> Flow:
> >> arp,in_port=61,vlan_tci=0x,dl_src=fa:16:3e:5e:79:d9,dl_dst=ff:ff:f
> >> f:ff:ff:ff,arp_spa=10.176.3.123,arp_tpa=10.176.2.19,arp_op=1,arp_sha=f
> >> a:16:3e:5e:79:d9,arp_tha=00:00:00:00:00:00
> >>
> >>
> >> bridge("br-int")
> >>
> >> 0. in_port=61, priority 100, cookie 0x862b95fc
> >> set_field:0x1->reg13
> >> set_field:0x7->reg11
> >> set_field:0x5->reg12
> >> set_field:0x1a->metadata
> >> set_field:0x4->reg14
> >> resubmit(,8)
> >> * Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7"
> >> (d516056b-19a6-4613-9838-8c62452fe31d)
> >> * Port Binding: logical_port "b19ceab1-c7fe-4c3b-8733-d88cabaa0a23",
> tunnel_key 4, chassis-name "383eb44a-de85-485a-9606-2fc649a9cbb9",
> chassis-str "os-compute-01"
> >> 8. reg14=0x4,metadata=0x1a,dl_src=fa:16:3e:5e:79:d9, priority 50,
> >> cookie 0x9a357820
> >> resubmit(,9)
> >> * Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7"
> >> (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
> >> * Logical flow: table=0 (ls_in_port_sec_l2), priority=50,
> >> match=(inport == "b19ceab1-c7fe-4c3b-8733-d88cabaa0a23" && eth.src ==
> >> {fa:16:3e:5e:79:d9}), actions=(next;)
> >>* Logical Switch Port: b19ceab1-c7fe-4c3b-8733-d88cabaa0a23 type
> >> (addresses ['fa:16:3e:5e:79:d9 10.176.3.123'], dynamic addresses [],
> >> security ['fa:16:3e:5e:79:d9 10.176.3.123'] 9. metadata=0x1a, priority
> >> 0, cookie 0x1a478ee1
> >> resubmit(,10)
> >> * Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7"
> >> (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
> >> * Logical flow: table=1 (ls_in_port_sec_ip), priority=0, match=(1),
> >> actions=(next;) 10.
> >> arp,reg14=0x4,metadata=0x1a,dl_src=fa:16:3e:5e:79:d9,arp_spa=10.176.3.
> >> 123,arp_sha=fa:16:3e:5e:79:d9, priority 90, cookie 0x8c5af8ff
> >> resubmit(,11)
> >> * Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7"
> >> (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
> >> * Logical flow: table=2 (ls_in_port_sec_nd), priority=90,
> >> match=(inport == "b19ceab1-c7fe-4c3b-8733-d88cabaa0a23" && eth.src ==
> >> fa:16:3e:5e:79:d9 && arp.sha == fa:16:3e:5e:79:d9 && arp.spa ==
> >> {10.176.3.123}),
[ovs-discuss] Questions about qinq packet get a wrong hash from 'skb_get_hash'
Hi all,
We find a problem in our our environment with ovs+linux-kernel-- The same
flow's packets will get different hash from 'skb_get_hash', if we push a vxlan
header to these packets, the outter udp's sprot will be different.
The call stack and corresponding skb info:
ovs_dp_process_packet --skb->data:a0c25fb502f2(mac header),
skb->network_header:a0c25fb50304(ip header), skb->protocol:0x8100,
skb->protocol:0x8100,, is_vlan_tag_present:true
ovs_vport_receive
netdev_port_receive
netdev_frame_hook skb->data:a0c25fb50300(inner vlan header),
skb->network_header:a0c25fb50300, skb->protocol:0x8100,
skb->protocol:0x8100, is_vlan_tag_present:true
__netif_receive_skb_core
__netif_receive_skb
netif_receive_skb_internal
napi_gro_receive
recv_one_pkt
hinic_rx_poll
hinic_poll
net_rx_action
__do_softirq
call_softirq
Problem Description:
When ovs received a qinq packet, the kernel has untagged the outer vlan and
save vlan_id in skb-> vlan_tci, vlan_protocol in skb->vlan_proto. Now
skb->protocol=0x8100,skb->vlan_protocol=0x8100 skb->.
In 'netdev_frame_hook': skb->data point to packet's inner vlan,
skb->network_header point to inner vlan too.
In 'ovs_vport_receive': After 'ovs_flow_key_extract', the skb->data will point
to mac header, skb->network_header will point to ip header. Then
'ovs_dp_process_packet' will call 'skb_get_hash'. 'skb_get_hash' will call
'__skb_flow_dissect' finally.
[cid:image001.jpg@01D60995.BC93D3B0]
In '__skb_flow_dissect'( "flow_dissector.c" line 1161 of newest kernel code) .
Because the network header is pointing to ip, when skb enter the vlan case in
the second time, it will extract vlan info in ip header, then
'__skb_flow_dissect' will return a error hash:
case htons(ETH_P_8021Q): {
const struct vlan_hdr *vlan = NULL;
struct vlan_hdr _vlan;
__be16 saved_vlan_tpid = proto;
if (dissector_vlan == FLOW_DISSECTOR_KEY_MAX &&
skb && skb_vlan_tag_present(skb)) {
proto = skb->protocol;
} else {
vlan = __skb_header_pointer(skb, nhoff,
sizeof(_vlan),
data, hlen, &_vlan);
if (!vlan) {
fdret = FLOW_DISSECT_RET_OUT_BAD;
break;
}
proto = vlan->h_vlan_encapsulated_proto;
nhoff += sizeof(*vlan);
}
Is there any useful patch to fix this bug? Or some suggestion?
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
