On Tue, 24 Jan 2017, Sean Bruno wrote:
On 01/24/17 08:27, Olivier Cochard-Labb?? wrote:
On Tue, Jan 24, 2017 at 3:17 PM, Sean Bruno <sbr...@freebsd.org
<mailto:sbr...@freebsd.org>> wrote:
Did you increase the number of rx/tx rings to 8 and the number of
descriptors to 4k in your tests or just the defaults?
Tuning are same as described in my previous email (rxd|txd=2048, rx|tx
process_limit=-1, max_interrupt_rate=16000).
[root@apu2]~# sysctl hw.igb.
hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: -1
hw.igb.num_queues: 0
hw.igb.header_split: 0
hw.igb.max_interrupt_rate: 16000
hw.igb.enable_msix: 1
hw.igb.enable_aim: 1
hw.igb.txd: 2048
hw.igb.rxd: 2048
Oh, I think you missed my note on these. In order to adjust txd/rxd you
need to tweak the iflib version of these numbers. nrxds/ntxds should be
adjust upwards to your value of 2048. nrxqs/ntxqs should be adjust
upwards to 8, I think, so you can test equivalent settings to the legacy
driver.
Specifically, you may want to adjust these:
dev.em.0.iflib.override_nrxds: 0
dev.em.0.iflib.override_ntxds: 0
dev.em.0.iflib.override_nrxqs: 0
dev.em.0.iflib.override_ntxqs: 0
That is painful.
My hack to increase the ifq length also no longer works:
X Index: if_em.c
X ===================================================================
X --- if_em.c (revision 312696)
X +++ if_em.c (working copy)
X @@ -1,3 +1,5 @@
X +int em_qlenadj = -1;
X +
-1 gives a null adjustment; 0 gives a default (very large ifq), and other
values give a non-null adustment.
X /*-
X * Copyright (c) 2016 Matt Macy <mm...@nextbsd.org>
X * All rights reserved.
X @@ -2488,7 +2490,10 @@
X
X /* Single Queue */
X if (adapter->tx_num_queues == 1) {
X - if_setsendqlen(ifp, scctx->isc_ntxd[0] - 1);
X + if (em_qlenadj == 0)
X + em_qlenadj = imax(2 * tick, 0) * 15 / 10;
X + // lem_qlenadj = imax(2 * tick, 0) * 42 / 100;
X + if_setsendqlen(ifp, scctx->isc_ntxd[0] + em_qlenadj);
X if_setsendqready(ifp);
X }
X
I don't want larger hardware queues, but sometimes want larger software
queues. ifq's used to give them. The if_setsenqlen() call is still there.
but no longer gives them.
The large queues are needed for backet blasting benchmarks since select()
doesn't work for udp sockets, so if the queues fill up then the benchmarks
must busy-wait or sleep waiting for them to drain, and timeout granularity
tends to prevent short sleeps from working so the queues run dry while
sleeping unless the queues are very large.
Bruce_______________________________________________
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
