Hi Raja,
For our tests we took today two new kvm compute nodes with vrouter and
without any Virtual Machines.
What we noticed is that if we spawn up to 4 VM within the same project and
Virtual Network on each compute node everything works all right.
If we add 5th VM on test compute node problem appears , and we can
reproduce this on different physical kvm compute nodes.
On this 5th Virtual machines we are starting seeing first SYN packet for
other VMs located on specific compute node (vrouter) .
All other packets from the transmission is normally delivered to target
virtual machines.
This is visible for all consecutive tcp flows between VM's, and only for
first packet from a new flow.
This 5th VM receives traffic for itself normally. So we end up in situation
where we have affected 4 VM's and one working without problem.
Problem could be resolved if we shutdown virtual port of this 5th VM or
delete it. After this step traffic for other vm is delivered normally.
If we create again new VM problem starts to be visible again.
We cannot see any new flows between client and 5th VM that receives
additional SYN packets that this VM shouldn't see.
All other flows are created successfully.
In the dropstat output we cannot see any changes especially related to the
drop counter.
Following lines you can find example of how we see trafic on this vm where
all first packets are sending
192.168.50.22 - client located on first compute node
192.168.50.27 - 192.168.50.38 - other vm with nginx, located on second
compute node
192.168.50.33 - VM where all first SYN packets are visible
[root@c-cn-28-10 centos]# ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP qlen 1000
inet 192.168.50.33/20 brd 10.189.63.255 scope global eth0
valid_lft forever preferred_lft forever
[root@c-cn-28-10 centos]# tcpdump "tcp[tcpflags] & (tcp-syn|tcp-ack) != 0"
and port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:20:22.626172 IP 192.168.50.22.51823 > 192.168.50.27.http: Flags [S], seq
2741907250, win 14600, options [mss 1460,sackOK,TS val 5010347 ecr
0,nop,wscale 7], length 0
14:20:23.631110 IP 192.168.50.22.60984 > 192.168.50.28.http: Flags [S], seq
3116446858, win 14600, options [mss 1460,sackOK,TS val 5011352 ecr
0,nop,wscale 7], length 0
14:20:24.637322 IP 192.168.50.22.46585 > 192.168.50.29.http: Flags [S], seq
3566871094, win 14600, options [mss 1460,sackOK,TS val 5012359 ecr
0,nop,wscale 7], length 0
14:20:25.643556 IP 192.168.50.22.41182 > 192.168.50.30.http: Flags [S], seq
1739097792, win 14600, options [mss 1460,sackOK,TS val 5013365 ecr
0,nop,wscale 7], length 0
14:20:26.651601 IP 192.168.50.22.36921 > 192.168.50.31.http: Flags [S], seq
3990314127, win 14600, options [mss 1460,sackOK,TS val 5014373 ecr
0,nop,wscale 7], length 0
14:20:27.659212 IP 192.168.50.22.40056 > 192.168.50.32.http: Flags [S], seq
1091797067, win 14600, options [mss 1460,sackOK,TS val 5015380 ecr
0,nop,wscale 7], length 0
14:20:28.665625 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [S], seq
1936544852, win 14600, options [mss 1460,sackOK,TS val 5016387 ecr
0,nop,wscale 7], length 0
14:20:28.665653 IP 192.168.50.33.http > 192.168.50.22.52838: Flags [S.],
seq 2591101890, ack 1936544853, win 14480, options [mss 1460,sackOK,TS val
4294865173 ecr 5016387,nop,wscale 7], length 0
14:20:28.666021 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [.], ack
1, win 115, options [nop,nop,TS val 5016388 ecr 4294865173], length 0
14:20:28.666046 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [P.],
seq 1:78, ack 1, win 115, options [nop,nop,TS val 5016388 ecr 4294865173],
length 77
14:20:28.666050 IP 192.168.50.33.http > 192.168.50.22.52838: Flags [.], ack
78, win 114, options [nop,nop,TS val 4294865173 ecr 5016388], length 0
14:20:28.666447 IP 192.168.50.33.http > 192.168.50.22.52838: Flags [P.],
seq 1:239, ack 78, win 114, options [nop,nop,TS val 4294865174 ecr
5016388], length 238
14:20:28.666947 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [.], ack
239, win 123, options [nop,nop,TS val 5016389 ecr 4294865174], length 0
14:20:28.666958 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [F.],
seq 78, ack 239, win 123, options [nop,nop,TS val 5016389 ecr 4294865174],
length 0
14:20:28.667002 IP 192.168.50.33.http > 192.168.50.22.52838: Flags [F.],
seq 239, ack 79, win 114, options [nop,nop,TS val 4294865174 ecr 5016389],
length 0
14:20:28.667340 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [.], ack
240, win 123, options [nop,nop,TS val 5016390 ecr 4294865174], length 0
14:20:28.671229 IP 192.168.50.22.46357 > 192.168.50.34.http: Flags [S], seq
2912677901, win 14600, options [mss 1460,sackOK,TS val 5016393 ecr
0,nop,wscale 7], length 0
14:20:29.679416 IP 192.168.50.22.53239 > 192.168.50.35.http: Flags [S], seq
1015724729, win 14600, options [mss 1460,sackOK,TS val 5017401 ecr
0,nop,wscale 7], length 0
14:20:30.687231 IP 192.168.50.22.40427 > 192.168.50.36.http: Flags [S], seq
3789917763, win 14600, options [mss 1460,sackOK,TS val 5018408 ecr
0,nop,wscale 7], length 0
14:20:31.693362 IP 192.168.50.22.47202 > 192.168.50.37.http: Flags [S], seq
2700187443, win 14600, options [mss 1460,sackOK,TS val 5019415 ecr
0,nop,wscale 7], length 0
14:20:32.699384 IP 192.168.50.22.37820 > 192.168.50.38.http: Flags [S], seq
2458579739, win 14600, options [mss 1460,sackOK,TS val 5020421 ecr
0,nop,wscale 7], length 0
Additioanl test with ping
[root@client]# ping 192.168.50.35 -c 2
PING 192.168.50.35 (192.168.50.35) 56(84) bytes of data.
64 bytes from 192.168.50.35: icmp_seq=2 ttl=64 time=0.498 ms
--- 192.168.50.35 ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 999ms
rtt min/avg/max/mdev = 0.498/0.498/0.498/0.000 ms
(on "the 5th vm" with interface address 192.168.50.33) tcpdump -i any icmp
-n
14:39:11.741773 IP 192.168.50.22 > 192.168.50.35: ICMP echo request, id
3626, seq 1, length 64
14:39:12.685724 IP 192.168.50.22 > 192.168.50.35: ICMP echo request, id
3627, seq 1, length 64
14:39:13.549434 IP 192.168.50.22 > 192.168.50.35: ICMP echo request, id
3628, seq 1, length 64
14:39:14.397740 IP 192.168.50.22 > 192.168.50.35: ICMP echo request, id
3629, seq 1, length 64
14:39:15.237451 IP 192.168.50.22 > 192.168.50.35: ICMP echo request, id
3630, seq 1, length 6
We are not seeing this icmp packets on the flow list
[destination compute node "flow -l"
28268 192.168.50.33:123 95.158.95.123:123 17 (1)
74012 95.158.95.123:123 192.168.50.33:123 17 (1)
238232<=>392568 192.168.50.33:22 10.163.14.57:47034 6 (1)
392568<=>238232 10.163.14.57:47034 192.168.50.33:22 6 (1)
Please let us know how to troubleshoot it in more details. What we can
check next.
Kind Regards
Piotr Pieprzycki
On 12 October 2015 at 19:17, Rajagopalan Sivaramakrishnan <[email protected]>
wrote:
> Hi Piotr,
> There isn’t a known issue in this respect. Can you please run
> “dropstats” on the destination compute node to see if any error counters
> are going up inside vrouter?
> Also, can you please run “flow –l” on the destination compute node to see
> if the flow action for the relevant flow is set to F (forward).
>
> Raja
>
> From: Users <[email protected]> on behalf of Piotr P <
> [email protected]>
> Date: Monday, October 12, 2015 at 7:39 AM
> To: "<[email protected]>" <[email protected]>
> Subject: [Users] Drops of the first SYN packet
>
>
> Hi All,
> We are experiencing problem with missing packets between Virtual Machines
> located within the same Virtual Network.
>
> Our setup Contrail 2.20 (buld 64), Priority of encapsulation: VxLAN,
> MPLSoUDP, MPLSoGRE
>
>
> The effect is visible during each new transmission/flow. There is missing
> first syn packet or first icmp ping.
> This affect each tcp session, it's delayed for syn retransmission time.
> Drops occurs for each subsequent new tcp session between VM's. We can
> repeat tcp connection between VM and first syn will be missing all the time.
> It's not visible in all Virtual Networks. There are some other Virtual
> Networks that works correctly.
> We can reproduce this behaviour with creating new networks with even only
> 2 virtual machines.
> Those networks differs only by other routing-target value.
>
> This effect
> does not appear when we have transmission outside VN (eg through physical
> gateway)
> it's only correlated when traffic stays within network.
>
> Tcpdump on compute nodes shows that:
> - All packets are visible on incoming tap interface (from source VM)
> - all packets are visible on bond interfaces on both compute nodes
> (vrouter) with VMs (bond active-passive toward physical switch )
> Computer nodes are located on the same physical switch (no visible
> errors on switch ports)
> - first packet/syn from transmission is missing on a tap interface toward
> destination VM
>
>
> Please let know if this is some known behaviour. What could be the steps
> to troubleshoot this issue?
>
> Kind Regards
> Piotr Pieprzycki
>
_______________________________________________
Users mailing list
[email protected]
http://lists.opencontrail.org/mailman/listinfo/users_lists.opencontrail.org
