I have a Enterprise T5220 server, running Solaris 10 that I am using as a backup server. On this server, I have a Layer 4, LACP-enabled link aggregation set up using two of the server's Gigabit NICs (e1000g2 and e1000g3) and until recently I was getting up to and sometimes over 1.5 Gb/s as desired. However, something has happened recently to where I can now barely get over 1 Gb/s. As far as I know, no patches were applied to the server and no changes were made to the switch that it's connected to (Nortel Passport 8600 Series) and the total amount of backup data sent to the server has stayed fairly constant. I have tried setting up the aggregation multiple times and in multiple ways to no avail. (LACP enabled/disabled, different policies, etc.) I've also tried using different ports on the server and switch to rule out any faulty port problems. Our networking guys assure me that the aggregation is set up correctly on the switch side but I can get more details if needed.
In order to attempt to better troubleshoot the problem, I run one of several network speed tools (nttcp, nepim, & iperf) as the "server" on the T5220, and I set up a spare X2100 as a "client". Both the server and client are connected to the same switch. The first set of tests with all three tools yields roughly 600 Mb/s. This seems a bit low to me, I seem to remember getting 700+ Mb/s prior to this "issue". When I run a second set of tests from two separate "client" X2100 servers, coming in on two different Gig ports on the T5220, each port also does ~600 Mb/s. I have also tried using crossover cables and I only get maybe a 50-75 Mb/s increase. After Googling Solaris network optimizations, I found that if I double tcp_max_buf to 2097152, and set tcp_xmit_hiwat & tcp_recv_hiwat to 524288, it bumps up the speed of a single Gig port to ~920 Mb/s. That's more like it! [b]Unfortunately however, even with the TCP tweaks enabled, I still only get a little over 1 Gb/s through the two aggregated Gig ports. It seems as though the aggregation is only using one port,[/b] though MRTG graphs of the two switch ports do in fact show that they are both being utilized equally, essentially splitting the 1 Gb/s speed between the two ports. Problem with the server? switch? Aggregation software? All the above? At any rate, I seem to be missing something.. Any help regarding this issue would be greatly appreciated! Regards, sundy Output of several commands on the T5220: uname -a: SunOS oitbus1 5.10 Generic_137111-07 sun4v sparc SUNW,SPARC-Enterprise-T5220 ifconfig -a (IP and broadcast hidden for security): lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 aggr1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet x.x.x.x netmask ffffff00 broadcast x.x.x.x ether 0:14:4f:ec:bc:1e dladm show-dev: e1000g0 link: unknown speed: 0 Mbps duplex: half e1000g1 link: unknown speed: 0 Mbps duplex: half e1000g2 link: up speed: 1000 Mbps duplex: full e1000g3 link: up speed: 1000 Mbps duplex: full dladm show-link: e1000g0 type: non-vlan mtu: 1500 device: e1000g0 e1000g1 type: non-vlan mtu: 1500 device: e1000g1 e1000g2 type: non-vlan mtu: 1500 device: e1000g2 e1000g3 type: non-vlan mtu: 1500 device: e1000g3 aggr1 type: non-vlan mtu: 1500 aggregation: key 1 dladm show-aggr: key: 1 (0x0001) policy: L4 address: 0:14:4f:ec:bc:1e (auto) device address speed duplex link state e1000g2 0:14:4f:ec:bc:1e 1000 Mbps full up attached e1000g3 <unknown> 1000 Mbps full up attached dladm show-aggr -L: key: 1 (0x0001) policy: L4 address: 0:14:4f:ec:bc:1e (auto) LACP mode: active LACP timer: short device activity timeout aggregatable sync coll dist defaulted expired e1000g2 active short yes yes yes yes no no e1000g3 active short yes yes yes yes no no dladm show-aggr -s: key: 1 ipackets rbytes opackets obytes %ipkts %opkts Total 464982722061215050501612388529872161440848661 e1000g2 30677028844072327428231142100939796617960694 66.0 59.5 e1000g3 15821243372049177622000967520476 64822888149 34.0 40.5 -- This message posted from opensolaris.org
