Hi, On 2015-10-20, at 10:24, Ian Smith <smi...@nimnet.asn.au> wrote: > Actually, you want to set hw.acpi.cpu.cx_lowest=C1 instead.
Done. On 2015-10-19, at 17:55, Luigi Rizzo <ri...@iet.unipi.it> wrote: > On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars <l...@netapp.com> wrote: >> The only other sysctls in ixl(4) that look relevant are: >> >> hw.ixl.rx_itr >> The RX interrupt rate value, set to 8K by default. >> >> hw.ixl.tx_itr >> The TX interrupt rate value, set to 4K by default. >> > > yes those. raise to 20-50k and see what you get in > terms of ping latency. While ixl(4) talks about 8K and 4K, the defaults actually seem to be: hw.ixl.tx_itr: 122 hw.ixl.rx_itr: 62 Doubling those values *increases* flood ping latency to ~200 usec (from ~116 usec). Halving them to 62/31 decreases flood ping latency to ~50 usec, but still doesn't increase iperf throughput (still 2.8 Gb/s). Going to 31/16 further drops latency to 24 usec, with no change in throughput. (Looking at the "interrupt Moderation parameters" #defines in sys/dev/ixl/ixl.h it seems that ixl likes to have its irq rates specified with some weird divider scheme.) With 5/5 (which corresponds to IXL_ITR_100K), I get down to 16 usec. Unfortunately, throughput is then also down to about 2 Gb/s. One thing I noticed in top is that one queue irq is using quite a bit of CPU when I run iperf: 11 0 -92 - 0K 1152K CPU2 2 0:19 50.98% intr{irq293: ixl1:q2} 11 0 -92 - 0K 1152K WAIT 3 0:02 5.18% intr{irq294: ixl1:q3} 0 0 -92 0 0K 8944K - 25 0:01 1.07% kernel{ixl1 que} 11 0 -92 - 0K 1152K WAIT 1 0:01 0.00% intr{irq292: ixl1:q1} 11 0 -92 - 0K 1152K WAIT 0 0:00 0.00% intr{irq291: ixl1:q0} 0 0 -92 0 0K 8944K - 22 0:00 0.00% kernel{ixl1 adminq} 0 0 -92 0 0K 8944K - 31 0:00 0.00% kernel{ixl1 que} 0 0 -92 0 0K 8944K - 31 0:00 0.00% kernel{ixl1 que} 0 0 -92 0 0K 8944K - 31 0:00 0.00% kernel{ixl1 que} 11 0 -92 - 0K 1152K WAIT -1 0:00 0.00% intr{irq290: ixl1:aq} With 10G ix interfaces and a throughput of ~9Gb/s, the CPU load is much lower: 11 0 -92 - 0K 1152K WAIT 0 0:05 7.67% intr{irq274: ix0:que } 0 0 -92 0 0K 8944K - 27 0:00 0.29% kernel{ix0 que} 0 0 -92 0 0K 8944K - 10 0:00 0.00% kernel{ix0 linkq} 11 0 -92 - 0K 1152K WAIT 1 0:00 0.00% intr{irq275: ix0:que } 11 0 -92 - 0K 1152K WAIT 3 0:00 0.00% intr{irq277: ix0:que } 11 0 -92 - 0K 1152K WAIT 2 0:00 0.00% intr{irq276: ix0:que } 11 0 -92 - 0K 1152K WAIT 18 0:00 0.00% intr{irq278: ix0:link} 0 0 -92 0 0K 8944K - 0 0:00 0.00% kernel{ix0 que} 0 0 -92 0 0K 8944K - 0 0:00 0.00% kernel{ix0 que} 0 0 -92 0 0K 8944K - 0 0:00 0.00% kernel{ix0 que} Lars
signature.asc
Description: Message signed with OpenPGP using GPGMail