On Mon, 2015-05-25 at 16:19 -0700, Eric Dumazet wrote: > On Mon, 2015-05-25 at 18:44 -0400, John A. Sullivan III wrote: > > On Mon, 2015-05-25 at 15:38 -0700, Eric Dumazet wrote: > > > On Mon, 2015-05-25 at 18:22 -0400, John A. Sullivan III wrote: > > > > > > > 2) Why do we still not negotiate the 16MB buffer that we get when we are > > > > not using GRE? > > > > > > What exact NIC handles receive side ? > > > > > > If drivers allocate a full 4KB page to hold each frame, > > > plus sk_buff overhead, > > > then 32MB of kernel memory translates to 8MB of TCP window space. > > > > > > > > > > > > > > Hi, Eric. I'm not sure I understand the question or how to obtain the > > information you've requested. The receive side system has 48GB of RAM > > but that does not sound like what you are requesting. > > > > I suspect the behavior is a "protection mechanism", i.e., it is being > > calculated for good reason. When I set the buffer to 16MB manually in > > nuttcp, performance degraded so I assume I was overrunning something. I > > am still downloading the traces. > > > > But I'm still mystified by why this only affects GRE traffic. Thanks - > > GRE is quite expensive, some extra cpu load is needed. > > On receiver, can you please check what exact driver is loaded ? > > Is it igb, ixgbe, e1000e, i40e ? > > ethtool -i eth0 > > GRE has extra 28 bytes of encapsulation, this definitely can make skb a > little bit fat. TCP has very simple heuristics (using power of two > steps) and a 50% factor can be explained by this extra 28 bytes for some > particular driver. > > You could emulate this at the sender (without GRE) by reducing the mtu > for the route to your target. > > ip route add 192.x.y.z via <gateway> mtu 1450 > >
The receiver as well as the gateway is using igb: root@vserveringestst-01:~# ethtool -i eth0 driver: igb version: 3.2.10-k firmware-version: 1.4-3 bus-info: 0000:01:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes Changing the MTU does not show the same degradation as GRE: root@gwhq-1:~# ip route add 192.168.224.2 via 192.168.128.1 mtu 1476 root@gwhq-1:~# nuttcp -T 60 -i 10 192.168.224.2 connect failed: Connection timed out interval option only supported for client/server mode root@gwhq-1:~# nuttcp -T 60 -i 10 192.168.224.2 644.6875 MB / 10.00 sec = 540.7944 Mbps 0 retrans 1121.1875 MB / 10.00 sec = 940.5201 Mbps 0 retrans 1121.2500 MB / 10.00 sec = 940.5744 Mbps 0 retrans 1121.1250 MB / 10.00 sec = 940.4777 Mbps 0 retrans 1121.2500 MB / 10.00 sec = 940.5757 Mbps 0 retrans 1028.8750 MB / 10.00 sec = 863.0736 Mbps 0 retrans 6171.9375 MB / 60.70 sec = 852.9101 Mbps 5 %TX 12 %RX 0 retrans 80.27 msRTT CPU does not seem to be an issue from what I can see. The systems are all sitting at 98% idle and even checking individual CPUs shows no overload. Thanks - John -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
