On Mon, May 30, 2011 at 8:57 AM, Jonathan Morton <[email protected]> wrote: > If most of your clients are mobile, you should use a tcp congestion control > algorithm such as Westwood+ which is designed for the task. This is designed > to distinguish between congestion and random packet losses. It is much less > aggressive at filling buffers than the default CUBIC.
Not only are they mobile, the behavior might be considered like that of a "thin" client in the context of "tcp-thin" http://www.mjmwired.net/kernel/Documentation/networking/tcp-thin.txt So the servers have two different sorts of connections. There would be thousands of long-lived connections with only an occasional packet going back and forth. Those streams are on lossy mobile networks. Then there are several hundred very "fat" and fast connections moving a lot of data. Sometimes a client might change from a "thin" stream to a "thick" stream if it must collect a lot of content. So maybe westwood+ along with the "tcp-thin" settings in 2.6.38 might be a good idea. Looking at one server this morning, I have about 27,000 TCP connections in ESTABLISHED state. Many (most) of these are "thin" flows to devices that exchange a packet only occasionally. The server has 16 cores talking to 2 GigE NICs with 8 queues each. There is about 40 meg of traffic flowing into the server from the network and the outbound bandwidth is about 8 meg/sec. Much of that 8 meg (about 2.5 meg) is logging traffic going to a local log host via IPv6 + jumbo frames. The way this is configured is there are two NICs, eth0 and eth1 and three vlans, lets call them vlan 2 (front end traffic) vlan 3 (logging traffic) and vlan 4 (backend traffic). VLAN 2 would be configured on the NICs (eth0.2 and eth1.2) and then bonded using balance-xor using the layer2-3 xmit hash. This way a given flow should always hash to a given vlan interface on a particular NIC. So I have three bond interfaces talking to a multiqueue aware vlan driver. This allows one processor to handle log traffic while a different processor handles front-end traffic and another processor handing a backend transaction at the same time. The higher inbound to outbound ratio is backwards from your traditional server profile but that is because the traffic that comes in gets compressed and sent back out and it is mostly text and text compresses nicely. /proc/sys/net/ipv4/tcp_ecn is currently set to "2" meaning use ECN if I see ECN set from the other end but don't initiate connections with ECN set. As I never initiate connections to the clients, that really isn't an issue. The client always initiates the connection so if I see ECN, it will be used. So the exercise I am going through is trying to determine the best qdisc and where to put it. On the bond interfaces, on the vlan interfaces or on the NICs? Something simple like SFQ would probably work ok. I just want to make sure a single packet to the client can get through without a lot of delay in the face of a "fat" stream going somewhere else. Currently it is using the default mq qdisc: root@foo:~> tc -s qdisc show qdisc mq 0: dev eth0 root Sent 5038416030858 bytes 1039959904 pkt (dropped 0, overlimits 0 requeues 24686) rate 0bit 0pps backlog 0b 0p requeues 24686 qdisc mq 0: dev eth1 root Sent 1380477553077 bytes 2131951760 pkt (dropped 0, overlimits 0 requeues 2934) rate 0bit 0pps backlog 0b 0p requeues 2934 So there have been no packets dropped and there is no backlog and the path is clean all the way to the Internet without any congestion in my network (the path is currently about 5 times bigger than current bandwidth utilization and is 10GigE all the way from the switch to which the server is connected all the way to the Internet). Any congestion would be somewhere upstream from me. Suggestions? George _______________________________________________ Bloat mailing list [email protected] https://lists.bufferbloat.net/listinfo/bloat
