Your dropbox links 404 for me, at least. SW rings being faster makes some sense to me, since it's (sometimes) the case that HW offload isn't necessarily faster, but it is avoiding putting load on the CPU...
- Rich On Fri, Jun 27, 2014 at 10:13 PM, Kris Shannon <k...@shannon.id.au> wrote: > On 24 June 2014 23:19, Josef 'Jeff' Sipek <jef...@josefsipek.net> wrote: >> Does it speed up if you disable apix? (/etc/system: set apix_enable = 0) >> >> Jeff. > > On 24 June 2014 23:41, Dan McDonald <dan...@omniti.com> wrote: >> On Jun 24, 2014, at 8:59 AM, Richard Elling via illumos-discuss >> <discuss@lists.illumos.org> wrote: >>> Kris, >>> are deep C-states enabled? >> >> If they are, disable them (via the BIOS). >> >> You mention a CPU difference between your two boxes. Are they the same >> socket-type and chipset as well? >> >> Dan > > Sorry for the slow response. It's been a hectic couple of days, but > I've finally been able to get back to this. > > There is nothing that explicitly mentions C-states in the description > of any of the bios settings, but the power management options are all > on the 'screw enenrgy efficiency give me more power' setting. > > I've set apix_enable = 0 and that has seemed to help the transmit side > of the aggregate. > > I'm still left with receive hovering around 500 Mbit/s. Excerpt from iperf: > > [ 3] 1270.0-1275.0 sec 275 MBytes 461 Mbits/sec > [ 3] 1275.0-1280.0 sec 296 MBytes 496 Mbits/sec > [ 3] 1280.0-1285.0 sec 264 MBytes 442 Mbits/sec > [ 3] 1285.0-1290.0 sec 292 MBytes 491 Mbits/sec > [ 3] 1290.0-1295.0 sec 171 MBytes 286 Mbits/sec > [ 3] 1295.0-1300.0 sec 307 MBytes 515 Mbits/sec > [ 3] 1300.0-1305.0 sec 278 MBytes 467 Mbits/sec > [ 3] 1305.0-1310.0 sec 263 MBytes 441 Mbits/sec > [ 3] 1310.0-1315.0 sec 291 MBytes 488 Mbits/sec > [ 3] 1315.0-1320.0 sec 272 MBytes 456 Mbits/sec > [ 3] 1320.0-1325.0 sec 281 MBytes 471 Mbits/sec > > I did a quick flamegraph of the kernel stacks (after grepping to > remove unix`idle) > > https://dl.dropboxusercontent.com/content_link/EIqwX5RiX2QJTXiEyj4fP8cCvRvj4HZYktEYqSfE7aYJB9ombZSWhV45wyVhh826 > > On the box that is working it turns out that recieve is still slower > over an aggregate than direct, but still more than ten times better > than the problem box. Another excerpt from iperf: > > [ 3] 1270.0-1275.0 sec 4271 MBytes 7166 Mbits/sec > [ 3] 1275.0-1280.0 sec 4121 MBytes 6913 Mbits/sec > [ 3] 1280.0-1285.0 sec 3708 MBytes 6221 Mbits/sec > [ 3] 1285.0-1290.0 sec 3500 MBytes 5872 Mbits/sec > [ 3] 1290.0-1295.0 sec 3494 MBytes 5863 Mbits/sec > [ 3] 1295.0-1300.0 sec 3828 MBytes 6423 Mbits/sec > [ 3] 1300.0-1305.0 sec 3259 MBytes 5467 Mbits/sec > [ 3] 1305.0-1310.0 sec 3856 MBytes 6468 Mbits/sec > [ 3] 1310.0-1315.0 sec 3625 MBytes 6082 Mbits/sec > [ 3] 1315.0-1320.0 sec 3869 MBytes 6491 Mbits/sec > [ 3] 1320.0-1325.0 sec 3713 MBytes 6229 Mbits/sec > [ 3] 1325.0-1330.0 sec 4040 MBytes 6779 Mbits/sec > > A flamegraph of the kernel stacks on this box (I also grepped out > zfs`zio_execute and genunix`ioctl because this box is pretty much > always running at least one "zfs receive" from somewhere) > > https://dl.dropboxusercontent.com/content_link/NQl8k7n9TjW9YqwOsV2JHB8TWEh5cauALt44AmrNtkZMcIEZeyyejkmHfDhwLqQ8 > > Counterintuitively (at least to me) it seems to be going faster using > software rings instead of hardware rings. ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
