Hi Jesse, I'm sorry but i guess i have some differences here.
>>GRO is simply best effort aggregation and does not wait to get a full 64k frame so when the packet is transmitted on the physical NIC, TSO will be limited what was aggregated. >> You're only guaranteed to get big TSO packets all the time when you generate them on the local machine like you did in your later tests. Yes I agree, its a software GRO ( best-effort) and that is why i do not see full 64k sized offloaded packets I did 10 sec iperf and tcpdump for full duration and recorded tcpdump data at both egress port at router namespace and external network interface on Network Node ( server IP- 1.1.1.34 and source SNAT IP- 1.1.4.92 ) [root@rhel7-23 ~]# ip netns exec qrouter-223d91ae-3025-4070-83d0-63faa58d819b tcpdump -ennqi qg-ab10ec1d-2b dst 1.1.1.34 > router_egress.log [root@rhel7-23 ~]# tcpdump -ennqi eth1 dst 1.1.1.34 > external_nic.log [root@rhel7-23 ~]# cat router_egress.log | head -n 30 21:35:07.628732 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 74: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 0 21:35:07.629839 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 66: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 0 21:35:07.629854 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 90: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 24 21:35:07.629859 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582 21:35:07.630609 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 7056: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 6990 21:35:07.630635 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 9852: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 9786 21:35:07.631272 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 8454: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 8388 21:35:07.631290 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 8454: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 8388 21:35:07.631974 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 8454: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 8388 21:35:07.631993 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582 21:35:07.632680 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194 21:35:07.632694 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776 21:35:07.633215 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776 21:35:07.633240 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194 21:35:07.633784 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582 21:35:07.633805 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 8454: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 8388 21:35:07.634372 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 21036: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 20970 21:35:07.634401 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194 21:35:07.634971 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582 21:35:07.634992 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582 21:35:07.635546 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194 21:35:07.635560 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776 21:35:07.635577 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194 21:35:07.636120 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582 21:35:07.636142 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582 21:35:07.636707 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194 21:35:07.636722 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194 21:35:07.636731 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 12648: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 12582 21:35:07.636749 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 4260: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 4194 21:35:07.637250 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 9852: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 9786 As you said, its best effort and so i do not see all 64k sized offloaded packets everytime. However, majority of packets are offloaded at egress port of Router Namespace. [root@rhel7-23 ~]# cat router_egress.log | grep "length 1464" | wc -l 1448 [root@rhel7-23 ~]# cat router_egress.log | wc -l 119360 So out of 119360 packets captured only 1448 packets were actually not at all offloaded ( Ethernet Frame of size 1464 ) However, when i take the same stats for external network interface [root@rhel7-23 ~]# cat external_nic.log | head -n 200 21:35:07.641752 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641753 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641754 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641755 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641755 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641756 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641757 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641758 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641758 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641759 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641761 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641761 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641762 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641763 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641764 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641765 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641766 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641767 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641769 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641769 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641770 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641771 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641772 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641773 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641774 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641775 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641776 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.641776 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.642116 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776 21:35:07.642223 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776 21:35:07.642452 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776 21:35:07.642743 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776 21:35:07.642764 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776 21:35:07.643112 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 16842: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 16776 21:35:07.643301 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.643304 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.643305 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.643306 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.643308 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.643308 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.643309 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 21:35:07.643310 fa:16:3e:86:37:b2 > 34:40:b5:c8:0a:3c, IPv4, length 1464: 1.1.4.92.59193 > 1.1.1.34.5001: tcp 1398 There are only a handful of offloaded packets [root@rhel7-23 ~]# cat external_nic.log | wc -l 4519467 [root@rhel7-23 ~]# cat external_nic.log | grep "length 1464" | wc -l 4504287 So out of 4519467 packets captured during full iperf duration, 4519467 packets actually had no offloading at all !! Is this the way we would expect it to work? The way i understood was that I should have seen a lot more offloaded packets ( not necessarily 64k sized but more than 1464 sized Ethernet frames ) on External Network Interface ( similar to the packets we see on egress port of router namespace ) Please rectify me if there is there something i misunderstood? Thanks and regards, Piyush Raman From: Jesse Gross <[email protected]> To: Piyush R Srivastava1/India/IBM@IBMIN Cc: [email protected], Mala Anand <[email protected]> Date: 12/19/2015 09:26 PM Subject: Re: Fw: [ovs-discuss] TSO not working for VM iperf data packets. On Sat, Dec 19, 2015 at 7:26 AM, Piyush R Srivastava1 <[email protected]> wrote: Hi, One other thing i noticed is that... For starting I DO SEE offloaded packets... However, after a few initial packets, Offloading seems to be reduced drastically and i see purely 1464 sized Ethernet Frames. [...] This seem to suggests that the issue is for the packets that are getting encapsulated at compute node and decapsulated at Network Node only We are using software GRO patch mentioned here at Network Nodes- http://openvswitch.org/pipermail/dev/2015-August/059335.html Does this info help in debugging the issue? There is no issue then. GRO is simply best effort aggregation and does not wait to get a full 64k frame so when the packet is transmitted on the physical NIC, TSO will be limited what was aggregated. You're only guaranteed to get big TSO packets all the time when you generate them on the local machine like you did in your later tests.
_______________________________________________ discuss mailing list [email protected] http://openvswitch.org/mailman/listinfo/discuss
