I'm trying to aggregate 10GbE ports for a realtime data acquisition application. I need to be able to get consistently high throughput to satisfy my application's demands.
I have a dual-X58 host with a MT26448 on each IOH. I can acquire > 39Gb/s in aggregate across four TCP streams (one per port). This throughput remains consistent across many consecutive netperf TCP_STREAM trials, but trying the same experiment hours or days later, I'll find that the performance suffers (< 30Gb/s). irq_balancer is disabled and all of the IRQs' affinity masks for one card are set to the same core, the other card's IRQs are set to a core on the other socket. I'm using 9000 byte MTUs. The majority of the system settings are at their defaults. I'm keeping the task affinity among the netserver processes static as well. I don't see any error messages from mlx* in dmesg. Can anyone suggest some parameters that might be able to help me find consistently high performance? ~~~~ $ sudo ethtool -i eth16 driver: mlx4_en (MT_0BD0110010) version: 1.4.2 (Sep 24 2009) firmware-version: 2.6.0 bus-info: 0000:28:00.0 $ lsb_release -a LSB Version: core-2.0-noarch:core-3.2-noarch:core-2.0-x86_64:core-3.2-x86_64:desktop-3.2-amd64:desktop-3.2-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch Distributor ID: SUSE LINUX Description: openSUSE 11.1 (x86_64) Release: 11.1 Codename: n/a $ cat /proc/version Linux version 2.6.29.6-9-debug (r...@z800-revo-0) (gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) ) #1 SMP Tue Sep 15 11:02:15 CDT 2009 $ sudo grep . /sys/module/mlx4_en/parameters/* /sys/module/mlx4_en/parameters/inline_thold:104 /sys/module/mlx4_en/parameters/ip_reasm:1 /sys/module/mlx4_en/parameters/num_lro:32 /sys/module/mlx4_en/parameters/pfcrx:0 /sys/module/mlx4_en/parameters/pfctx:0 /sys/module/mlx4_en/parameters/rss_mask:5 /sys/module/mlx4_en/parameters/rss_xor:0 $ sudo grep . /sys/module/mlx4_core/parameters/* /sys/module/mlx4_core/parameters/block_loopback:1 /sys/module/mlx4_core/parameters/debug_level:0 /sys/module/mlx4_core/parameters/enable_qos:N /sys/module/mlx4_core/parameters/internal_err_reset:1 /sys/module/mlx4_core/parameters/log_mtts_per_seg:3 /sys/module/mlx4_core/parameters/log_num_cq:0 /sys/module/mlx4_core/parameters/log_num_mac:2 /sys/module/mlx4_core/parameters/log_num_mcg:0 /sys/module/mlx4_core/parameters/log_num_mpt:0 /sys/module/mlx4_core/parameters/log_num_mtt:0 /sys/module/mlx4_core/parameters/log_num_qp:0 /sys/module/mlx4_core/parameters/log_num_srq:0 /sys/module/mlx4_core/parameters/log_num_vlan:0 /sys/module/mlx4_core/parameters/log_rdmarc_per_qp:0 /sys/module/mlx4_core/parameters/msi_x:1 /sys/module/mlx4_core/parameters/set_4k_mtu:0 /sys/module/mlx4_core/parameters/use_prio:N -- -Brian -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
