Hi Piotr,
I didn't fully understand your description below, so I’ll describe the
expected behavior. The first packet of the flow should be sent to the vrouter
agent over the pkt0 interface. Subsequently, the agent should create a flow
with the flow action set to forward (which should be visible in “flow –l”). For
the packet that gets dropped, do you see it in “tcpdump –ni pkt0 –x” (you might
get some messages from tcpdump indicating bad header length, but you can ignore
those) and “flow –l". Can you please check this on the compute node where you
see the SYN being dropped? Are you seeing issues with ping too or is it only
with TCP SYN?
Raja
From: Piotr P <[email protected]<mailto:[email protected]>>
Date: Tuesday, October 13, 2015 at 7:53 AM
To: Rajagopalan Sivaramakrishnan <[email protected]<mailto:[email protected]>>
Cc: "<[email protected]<mailto:[email protected]>>"
<[email protected]<mailto:[email protected]>>
Subject: Re: [Users] Drops of the first SYN packet
Hi Raja,
For our tests we took today two new kvm compute nodes with vrouter and without
any Virtual Machines.
What we noticed is that if we spawn up to 4 VM within the same project and
Virtual Network on each compute node everything works all right.
If we add 5th VM on test compute node problem appears , and we can reproduce
this on different physical kvm compute nodes.
On this 5th Virtual machines we are starting seeing first SYN packet for other
VMs located on specific compute node (vrouter) .
All other packets from the transmission is normally delivered to target virtual
machines.
This is visible for all consecutive tcp flows between VM's, and only for first
packet from a new flow.
This 5th VM receives traffic for itself normally. So we end up in situation
where we have affected 4 VM's and one working without problem.
Problem could be resolved if we shutdown virtual port of this 5th VM or delete
it. After this step traffic for other vm is delivered normally.
If we create again new VM problem starts to be visible again.
We cannot see any new flows between client and 5th VM that receives additional
SYN packets that this VM shouldn't see.
All other flows are created successfully.
In the dropstat output we cannot see any changes especially related to the drop
counter.
Following lines you can find example of how we see trafic on this vm where all
first packets are sending
192.168.50.22 - client located on first compute node
192.168.50.27 - 192.168.50.38 - other vm with nginx, located on second compute
node
192.168.50.33 - VM where all first SYN packets are visible
[root@c-cn-28-10 centos]# ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
inet 127.0.0.1/8<http://127.0.0.1/8> scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP
qlen 1000
inet 192.168.50.33/20<http://192.168.50.33/20> brd 10.189.63.255 scope
global eth0
valid_lft forever preferred_lft forever
[root@c-cn-28-10 centos]# tcpdump "tcp[tcpflags] & (tcp-syn|tcp-ack) != 0" and
port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:20:22.626172 IP 192.168.50.22.51823 > 192.168.50.27.http: Flags [S], seq
2741907250, win 14600, options [mss 1460,sackOK,TS val 5010347 ecr 0,nop,wscale
7], length 0
14:20:23.631110 IP 192.168.50.22.60984 > 192.168.50.28.http: Flags [S], seq
3116446858, win 14600, options [mss 1460,sackOK,TS val 5011352 ecr 0,nop,wscale
7], length 0
14:20:24.637322 IP 192.168.50.22.46585 > 192.168.50.29.http: Flags [S], seq
3566871094, win 14600, options [mss 1460,sackOK,TS val 5012359 ecr 0,nop,wscale
7], length 0
14:20:25.643556 IP 192.168.50.22.41182 > 192.168.50.30.http: Flags [S], seq
1739097792, win 14600, options [mss 1460,sackOK,TS val 5013365 ecr 0,nop,wscale
7], length 0
14:20:26.651601 IP 192.168.50.22.36921 > 192.168.50.31.http: Flags [S], seq
3990314127, win 14600, options [mss 1460,sackOK,TS val 5014373 ecr 0,nop,wscale
7], length 0
14:20:27.659212 IP 192.168.50.22.40056 > 192.168.50.32.http: Flags [S], seq
1091797067, win 14600, options [mss 1460,sackOK,TS val 5015380 ecr 0,nop,wscale
7], length 0
14:20:28.665625 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [S], seq
1936544852, win 14600, options [mss 1460,sackOK,TS val 5016387 ecr 0,nop,wscale
7], length 0
14:20:28.665653 IP 192.168.50.33.http > 192.168.50.22.52838: Flags [S.], seq
2591101890, ack 1936544853, win 14480, options [mss 1460,sackOK,TS val
4294865173 ecr 5016387,nop,wscale 7], length 0
14:20:28.666021 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [.], ack 1,
win 115, options [nop,nop,TS val 5016388 ecr 4294865173], length 0
14:20:28.666046 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [P.], seq
1:78, ack 1, win 115, options [nop,nop,TS val 5016388 ecr 4294865173], length 77
14:20:28.666050 IP 192.168.50.33.http > 192.168.50.22.52838: Flags [.], ack 78,
win 114, options [nop,nop,TS val 4294865173 ecr 5016388], length 0
14:20:28.666447 IP 192.168.50.33.http > 192.168.50.22.52838: Flags [P.], seq
1:239, ack 78, win 114, options [nop,nop,TS val 4294865174 ecr 5016388], length
238
14:20:28.666947 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [.], ack
239, win 123, options [nop,nop,TS val 5016389 ecr 4294865174], length 0
14:20:28.666958 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [F.], seq
78, ack 239, win 123, options [nop,nop,TS val 5016389 ecr 4294865174], length 0
14:20:28.667002 IP 192.168.50.33.http > 192.168.50.22.52838: Flags [F.], seq
239, ack 79, win 114, options [nop,nop,TS val 4294865174 ecr 5016389], length 0
14:20:28.667340 IP 192.168.50.22.52838 > 192.168.50.33.http: Flags [.], ack
240, win 123, options [nop,nop,TS val 5016390 ecr 4294865174], length 0
14:20:28.671229 IP 192.168.50.22.46357 > 192.168.50.34.http: Flags [S], seq
2912677901, win 14600, options [mss 1460,sackOK,TS val 5016393 ecr 0,nop,wscale
7], length 0
14:20:29.679416 IP 192.168.50.22.53239 > 192.168.50.35.http: Flags [S], seq
1015724729, win 14600, options [mss 1460,sackOK,TS val 5017401 ecr 0,nop,wscale
7], length 0
14:20:30.687231 IP 192.168.50.22.40427 > 192.168.50.36.http: Flags [S], seq
3789917763, win 14600, options [mss 1460,sackOK,TS val 5018408 ecr 0,nop,wscale
7], length 0
14:20:31.693362 IP 192.168.50.22.47202 > 192.168.50.37.http: Flags [S], seq
2700187443, win 14600, options [mss 1460,sackOK,TS val 5019415 ecr 0,nop,wscale
7], length 0
14:20:32.699384 IP 192.168.50.22.37820 > 192.168.50.38.http: Flags [S], seq
2458579739, win 14600, options [mss 1460,sackOK,TS val 5020421 ecr 0,nop,wscale
7], length 0
Additioanl test with ping
[root@client]# ping 192.168.50.35 -c 2
PING 192.168.50.35 (192.168.50.35) 56(84) bytes of data.
64 bytes from 192.168.50.35<http://192.168.50.35>: icmp_seq=2 ttl=64 time=0.498
ms
--- 192.168.50.35 ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 999ms
rtt min/avg/max/mdev = 0.498/0.498/0.498/0.000 ms
(on "the 5th vm" with interface address 192.168.50.33) tcpdump -i any icmp -n
14:39:11.741773 IP 192.168.50.22 > 192.168.50.35<http://192.168.50.35>: ICMP
echo request, id 3626, seq 1, length 64
14:39:12.685724 IP 192.168.50.22 > 192.168.50.35<http://192.168.50.35>: ICMP
echo request, id 3627, seq 1, length 64
14:39:13.549434 IP 192.168.50.22 > 192.168.50.35<http://192.168.50.35>: ICMP
echo request, id 3628, seq 1, length 64
14:39:14.397740 IP 192.168.50.22 > 192.168.50.35<http://192.168.50.35>: ICMP
echo request, id 3629, seq 1, length 64
14:39:15.237451 IP 192.168.50.22 > 192.168.50.35<http://192.168.50.35>: ICMP
echo request, id 3630, seq 1, length 6
We are not seeing this icmp packets on the flow list
[destination compute node "flow -l"
28268 192.168.50.33:123<http://192.168.50.33:123>
95.158.95.123:123<http://95.158.95.123:123> 17 (1)
74012 95.158.95.123:123<http://95.158.95.123:123>
192.168.50.33:123<http://192.168.50.33:123> 17 (1)
238232<=>392568 192.168.50.33:22<http://192.168.50.33:22>
10.163.14.57:47034<http://10.163.14.57:47034> 6 (1)
392568<=>238232 10.163.14.57:47034<http://10.163.14.57:47034>
192.168.50.33:22<http://192.168.50.33:22> 6 (1)
Please let us know how to troubleshoot it in more details. What we can check
next.
Kind Regards
Piotr Pieprzycki
On 12 October 2015 at 19:17, Rajagopalan Sivaramakrishnan
<[email protected]<mailto:[email protected]>> wrote:
Hi Piotr,
There isn’t a known issue in this respect. Can you please run “dropstats”
on the destination compute node to see if any error counters are going up
inside vrouter?
Also, can you please run “flow –l” on the destination compute node to see if
the flow action for the relevant flow is set to F (forward).
Raja
From: Users
<[email protected]<mailto:[email protected]>>
on behalf of Piotr P <[email protected]<mailto:[email protected]>>
Date: Monday, October 12, 2015 at 7:39 AM
To: "<[email protected]<mailto:[email protected]>>"
<[email protected]<mailto:[email protected]>>
Subject: [Users] Drops of the first SYN packet
Hi All,
We are experiencing problem with missing packets between Virtual Machines
located within the same Virtual Network.
Our setup Contrail 2.20 (buld 64), Priority of encapsulation: VxLAN, MPLSoUDP,
MPLSoGRE
The effect is visible during each new transmission/flow. There is missing first
syn packet or first icmp ping.
This affect each tcp session, it's delayed for syn retransmission time.
Drops occurs for each subsequent new tcp session between VM's. We can repeat
tcp connection between VM and first syn will be missing all the time.
It's not visible in all Virtual Networks. There are some other Virtual Networks
that works correctly.
We can reproduce this behaviour with creating new networks with even only 2
virtual machines.
Those networks differs only by other routing-target value.
This effect
does not appear when we have transmission outside VN (eg through physical
gateway)
it's only correlated when traffic stays within network.
Tcpdump on compute nodes shows that:
- All packets are visible on incoming tap interface (from source VM)
- all packets are visible on bond interfaces on both compute nodes (vrouter)
with VMs (bond active-passive toward physical switch )
Computer nodes are located on the same physical switch (no visible errors on
switch ports)
- first packet/syn from transmission is missing on a tap interface toward
destination VM
Please let know if this is some known behaviour. What could be the steps to
troubleshoot this issue?
Kind Regards
Piotr Pieprzycki
_______________________________________________
Users mailing list
[email protected]
http://lists.opencontrail.org/mailman/listinfo/users_lists.opencontrail.org
