Hi, Here is another set-up where we can see the problem: client and server are in compute1 and SF is in compute2.
Compute1 has IP 192.168.2.3 Compute2 has IP 192.168.2.4 If we tcpdump what goes from compute2 to compute1 and while sniffing we send traffic through the chain, this is what we get: 12:58:15.517688 ARP, Request who-has 192.168.2.4 tell 192.168.2.3, length 28 12:58:15.517707 ARP, Reply 192.168.2.4 is-at 52:54:00:9f:0c:76 (oui Unknown), length 28 12:58:20.507515 IP 192.168.2.3.36551 > 192.168.2.4.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 97: openflow:93452943833469 12:58:20.507640 IP 192.168.2.4.60213 > 192.168.2.3.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 99: openflow:119801980855940 12:58:25.508796 IP 192.168.2.3.36551 > 192.168.2.4.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 97: openflow:93452943833469 12:58:25.510184 IP 192.168.2.4.60213 > 192.168.2.3.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 99: openflow:119801980855940 12:58:30.066510 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 82 12:58:30.069096 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 74 12:58:30.069263 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 113 12:58:30.101390 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 74 12:58:30.118118 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 122 12:58:30.122351 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 1322 12:58:30.122754 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 162 12:58:30.166343 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 74 12:58:30.507560 IP 192.168.2.3.36551 > 192.168.2.4.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 97: openflow:93452943833469 12:58:30.508179 IP 192.168.2.4.60213 > 192.168.2.3.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 99: openflow:119801980855940 12:58:32.447144 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 74 12:58:32.450561 IP 192.168.2.4.48926 > 192.168.2.3.6633: UDP, length 74 12:58:35.508493 IP 192.168.2.4.60213 > 192.168.2.3.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 99: openflow:119801980855940 12:58:35.508542 IP 192.168.2.3.36551 > 192.168.2.4.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 97: openflow:93452943833469 12:58:40.508362 IP 192.168.2.4.60213 > 192.168.2.3.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 99: openflow:119801980855940 12:58:40.512571 IP 192.168.2.3.36551 > 192.168.2.4.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 97: openflow:93452943833469 12:58:45.513612 IP 192.168.2.4.60213 > 192.168.2.3.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 99: openflow:119801980855940 Again, suddenly there are VXLAN packets using destination port 6633 even though they exit through the vxlan port. Here are the flows: Compute1: http://pastebin.com/pShGfEkn Compute2: http://pastebin.com/E9s9sY64 Ovs-ofctl show br-int int Compute2: http://pastebin.com/N7H3bHur Note in compute2 that in table=111, packets are egressing using the following flow: table=111, n_packets=22, n_bytes=3770, tun_id=0x12,dl_dst=fa:16:3e:48:61:7c actions=output:5 The mac address of the server is fa:16:3e:48:61:7c. You can see that output:5 is: 5(vxlan-192.168.2): addr:e2:7c:cf:8e:69:c5 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max Please tell me if you need more information. Regards, Manuel From: Yang, Yi Y [mailto:[email protected]] Sent: Wednesday, November 23, 2016 1:32 PM To: Manuel Buil <[email protected]> Cc: [email protected]; Georgios Paraskevopoulos ([email protected]) <[email protected]> Subject: RE: Potential bug in NSH patch for OVS Manuel, can you send the openflow entry for this output? If it is an explicit output to vxlan port, vxlan module will set tp_dst to vxlan's tp_dst. In dump-flows you provided, there was't any packet hitting that flow entry. From: Manuel Buil [mailto:[email protected]] Sent: Wednesday, November 23, 2016 8:24 PM To: Yang, Yi Y <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; Georgios Paraskevopoulos ([email protected]<mailto:[email protected]>) <[email protected]<mailto:[email protected]>> Subject: RE: Potential bug in NSH patch for OVS Hi Yi Yang, Compute1 has both ports, vxlangpe controlled by SFC and vxlan controlled by netvirt. The packets egress through vxlan port but using the destination port 6633. They don't egress through vxlan-gpe. /Manuel From: Yang, Yi Y [mailto:[email protected]] Sent: Wednesday, November 23, 2016 1:20 PM To: Manuel Buil <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; Georgios Paraskevopoulos ([email protected]<mailto:[email protected]>) <[email protected]<mailto:[email protected]>> Subject: RE: Potential bug in NSH patch for OVS Then the SFF in compute1 should have two vlxan ports, one (vxlangpe) is controlled by SFC, another (vxlan) is controlled by netvirt, so it should be SFC's issue but not ovs' issue. From: Manuel Buil [mailto:[email protected]] Sent: Wednesday, November 23, 2016 4:34 PM To: Yang, Yi Y <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; Georgios Paraskevopoulos ([email protected]<mailto:[email protected]>) <[email protected]<mailto:[email protected]>> Subject: RE: Potential bug in NSH patch for OVS Hi Yi Yang, I think we have different views here. The packet was already processed by the SFC pipeline and now it is again in the Netvirt pipeline. Why should then the packet be sent using the vxlan-gpe interface from the compute? The packet should use the vxlan interface of Netvirt which is what actually happens. The problem is that OVS confuses the port and sends it with port 6633 instead of 4789. /Manuel From: Yang, Yi Y [mailto:[email protected]] Sent: Wednesday, November 23, 2016 1:21 AM To: Manuel Buil <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; Georgios Paraskevopoulos ([email protected]<mailto:[email protected]>) <[email protected]<mailto:[email protected]>> Subject: RE: Potential bug in NSH patch for OVS I think that is NetVirt classifier issue, in sfc103 demo, we have on SF use case. In your case, you must have two classifiers, one is in compute1, another is in compute2, for compute1, I think you have enabled NetVirt to send back the packet to the client, but for compute2, NetVirt can't support this, right? Classifier2 must have VxLAN-gpe port for this in compute2. From: Manuel Buil [mailto:[email protected]] Sent: Wednesday, November 23, 2016 1:12 AM To: Yang, Yi Y <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; Georgios Paraskevopoulos ([email protected]<mailto:[email protected]>) <[email protected]<mailto:[email protected]>> Subject: Potential bug in NSH patch for OVS Hi Yi Yang, I am observing that sometimes our SFC OPNFV test case fails. I am trying to investigate why and I think I found the problem. ### SET-UP ### The environment is the following: the http traffic from client gets classified and assigned to a chain with one SF. After the traffic visits that SF, the SFC encapsulation is removed and the packet is sent to the netvirt pipeline. The scenario that fails is the following: client and SF are in compute1, the server is in compute2. ### PROBLEM ### All the SFC process happens in one OVS because both classifier and SF are in that OVS. Then the packet leaves SFC pipeline and gets into Netvirt pipeline which egress the packet using the vxlan tunnel it has to connect to the compute2. The problem is that instead of using udp port 4789 as it should, it uses udp port 6633. Compute2 has no SFC service so there is no vxlan port listening on udp port 6633 and thus the packet is dropped. I captured the packet using ovs-dpctl and everything is correct except the tp_dst which is 6633 even though it should be 4789: recirc_id(0),nsh(nsi=254,nsp=3c,nsh_mdtype=1,nsh_np=3,nshc1=c0a80006,nshc2=12),tunnel(tun_id=0x3c,src=11.0.0.6,dst=10.20.0.11,vxlan(gpe(np=0x3c)),flags(-df+csum+key)),in_port(8),skb_mark(0),eth(src=fa:16:3e:6b:08:c7,dst=fa:16:3e:4a:be:0e),eth_type(0x0800),ipv4(dst=11.0.0.4/255.255.255.254,tos=0/0x3,frag=no), packets:0, bytes:0, used:never, actions:set(tunnel(tun_id=0x12,src=192.168.2.4,dst=192.168.2.3,ttl=64,tp_src=49676,tp_dst=6633,vxlan(gpe(np=0x4,flags=0)),flags(df|csum|key))),pop_nsh,4 These are the ports (you can see that it egresses on 4 ==> vxlan_sys_4789): port 0: ovs-system (internal) port 1: br-ex (internal) port 2: p_aec4d661-0 (internal) port 3: br-int (internal) port 4: vxlan_sys_4789 (vxlan) port 5: tapbbd70ea7-7b port 6: tapd5cbb645-01 port 7: tap3f9a7479-51 port 8: vxlan_sys_6633 (vxlan) If I sniff the packets, I can see that all the traffic coming from compute1 and going to compute2 is sent through vxlan and udp port 4789, except for the packets that traversed SFC, which are sent with port 6633: 16:30:45.508076 IP 192.168.2.4.60213 > 192.168.2.3.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 99: openflow:119801980855940 16:30:45.508288 IP 192.168.2.3.36551 > 192.168.2.4.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 97: openflow:93452943833469 16:30:46.356412 IP 192.168.2.4.52024 > 192.168.2.3.6633: VXLAN, flags [I] (0x08), vni 18 IP 11.0.0.4.38652 > 11.0.0.5.http: Flags [S], seq 340357625, win 28200, options [mss 1410,sackOK,TS val 561484 ecr 0,nop,wscale 7], length 0 16:30:46.356679 IP 192.168.2.3 > 192.168.2.4: ICMP 192.168.2.3 udp port 6633 unreachable, length 118 16:30:47.354970 IP 192.168.2.4.52024 > 192.168.2.3.6633: VXLAN, flags [I] (0x08), vni 18 IP 11.0.0.4.38652 > 11.0.0.5.http: Flags [S], seq 340357625, win 28200, options [mss 1410,sackOK,TS val 561734 ecr 0,nop,wscale 7], length 0 16:31:20.508294 IP 192.168.2.3.36551 > 192.168.2.4.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 97: openflow:93452943833469 16:31:20.509431 IP 192.168.2.4.60213 > 192.168.2.3.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 99: openflow:119801980855940 16:31:25.507641 IP 192.168.2.3.36551 > 192.168.2.4.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 97: openflow:93452943833469 16:31:25.507728 IP 192.168.2.4.60213 > 192.168.2.3.4789: VXLAN, flags [I] (0x08), vni 0 LLDP, length 99: openflow:119801980855940 Can you investigate if this problem might be related to your NSH patch for OVS? Perhaps packets that traverse the SFC pipeline have L4 port destination hardcoded as 6633? Thanks, Manuel
_______________________________________________ sfc-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/sfc-dev
