Hi folks, I got a problem when I used the recent OVS version, openvswitch-2.12.0, not sure that the problem is related with the old version but also tested 2.13.0 just in case.
When using sudo command on Ubuntu 16.04, dns query is involved, where my problem began from. The DNS lookup send two UDP packets, one is for IPv4, the other IPv6. Some flows are added to do NAT on the VM's packets, which looks like overlay and underlay. I might think it works well with TCP traffic until now, but with 2 UDP packets at the nearly same time, one of them is dropped occasionally. At the beginng all UDP packets are forwarded to userspace by upcall because there is no flow in the kernel. After some of debugging, TWO upcall handler threads receive a packet one by one and treat the packet at the same time, which means there is a race condition to create a conntrack. Sometimes the conntrack is handled properly, but not perfectly. After changing n-handler-threads to 1, the problem is gone but set it to 2, coming back. I am not sure a TCP syn packet could be effected, but do not agree with because TCP just sends a syn packet for a connection at the same time. TCP syn pakcets on the different connections can be handled by the different threads. I might guess that if tcp packets with data received without the conntrack, they could be effected by this scenario. The strange thing that I’ve experienced is that if I add some of debugging code like printk() in ovs_packet_cmd_execute() in kernel to trace some of code, all is working properly. Here are the trace of conntrack: You can see 2 packets were involved on each conntrack. This is normal case without any drop. udp 17 59 src=1.1.1.111 dst=8.8.8.8 sport=54534 dport=53 packets=2 bytes=104 src=8.8.8.8 dst=172.31.250.0 sport=53 dport=54534 packets=2 bytes=254 [ASSURED] mark=1 secctx=system_u:object_r:unlabeled_t:s0 zone=65535 delta-time=0 use=1 udp 17 59 src=1.1.1.111 dst=8.8.8.8 sport=33157 dport=53 packets=2 bytes=104 src=8.8.8.8 dst=172.31.250.0 sport=53 dport=33157 packets=2 bytes=254 [ASSURED] mark=1 secctx=system_u:object_r:unlabeled_t:s0 zone=65535 delta-time=0 use=1 This is the case that a packet was dropped. udp 17 59 src=1.1.1.111 dst=8.8.8.8 sport=33288 dport=53 packets=1 bytes=52 src=8.8.8.8 dst=172.31.250.0 sport=53 dport=33288 packets=1 bytes=127 mark=1 secctx=system_u:object_r:unlabeled_t:s0 zone=65535 delta-time=1 use=1 Sometimes I saw the below logs warned me something is wrong. 2020-04-09T12:01:06.107Z|00002|dpif(handler315)|WARN|system@ovs-system: execute ct(commit,zone=65535,mark=0x6d/0xfff fffff,nat(src=172.24.177.33)),set(eth(dst=56:7d:45:28:d3:4f)),1 failed (Invalid argument) on packet udp,vlan_tci=0x0 000,dl_src=86:8b:0a:00:06:08,dl_dst=8e:72:0a:00:06:01,nw_src=10.0.6.8,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_ src=51528,tp_dst=53 udp_csum:ea88 with metadata skb_priority(0),skb_mark(0),ct_state(0x21),ct_zone(0xffff),ct_tuple4(src=10.0.6.8,dst=8.8.8.8,proto=1 7,tp_src=51528,tp_dst=53),in_port(3) mtu 0 Does anyone have experience like this? or know the way to solve it? Best Regards. Jiho Jung
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
