On Sun, 20 Jan 2019, Eugene Grosbein wrote:
19.01.2019 17:21, Bruce Evans wrote:
Your problem looks more like lost interrupts. All em NICs should interrupt
at the default interrupt moderation rate of 8 kHz under load. Once there
are are that many interrupts, there is not much else that can go wrong (nfs
would have to be working to generate that many interrupts).
I have a patch (in production since 8.x) that makes em(4) support
hw.em.max_interrupt_rate
just like igb(4) supports hw.igb.max_interrupt_rate:
http://www.grosbein.net/freebsd/patches/em_sysctl-11.0.diff.gz
It also brings in sysctls dev.em.X.max_interrupt_rate and
hw.em.max_interrupt_rate sets defaults for them.
This is inverted and spelled dev.em.X.itr for em.
Hmm, em already has this, but it is only a read-only tunable.
igb seems to have gone away. In FreeBSD-11, its dev.em.X.max_interrupt_rate
is also only a tunable.
I use the variants of the following fix for itr in FreeBSD-[7-13]
XX Index: if_em.c
XX ===================================================================
XX --- if_em.c (revision 332488)
XX +++ if_em.c (working copy)
XX @@ -908,10 +910,10 @@
XX E1000_REGISTER(hw, E1000_TADV),
XX em_tx_abs_int_delay_dflt);
XX em_add_int_delay_sysctl(adapter, "itr",
XX - "interrupt delay limit in usecs/4",
XX + "interrupt delay limit in usecs",
XX &adapter->tx_itr,
XX E1000_REGISTER(hw, E1000_ITR),
XX - DEFAULT_ITR);
XX + 1000000 / MAX_INTS_PER_SEC);
XX
XX hw->mac.autoneg = DO_AUTO_NEG;
XX hw->phy.autoneg_wait_to_complete = FALSE;
This fixes the description and the initial value for the sysctl to match
the code. The description almost matches the buggy initial value. The
hardware has power of 2 units, but the code scales to microseconds. Except
the initial value has was in hardware units scaled by another power of 2
which made them nearly microseconds/4. The code sets the initial value to
a representation of 125 usec (8 kHz), but the sysctl says that the initial
value is 488 and the description says that this is a representation of
488/4 = 122 usec. However, writing back this value using sysctl gives
488 usec (~2 kHz). The magic number 122 is 125 mis-scaled by 1000/1024.
FreeBSD[7-10] have lem in a separate file with the bug duplicated, so
need the patch duplicated. FreeBSD[7-8] don't have a sysctl for this.
They default to 125 usec and there is no way to see or change the value.
I usually want the smaller value of 0, and hard-code this when there is
no sysctl.
DEFAULT_ITR is used mainly to obfuscate this. IGB_DEFAULT_ITR and
IGB_LINK_ITR are also defined, but are not used even in versions of FreeBSD
that have igb.
I use hw.em.max_interrupt_rate=32000 for 1GB link passing average sized packets
(about 600 bytes per packet at average) but driver's default 8000 should be
nearly fine
for full size packets (1500 or above) and this 8000 limit cannot be reason for
such low throughput.
0 for itr maxes out at about 100 kHz here. This is good for low latency with
small packets.
My version of bge dynamically modifies the rate to match the rx load (no
moderation for light loads). tx is handled specially and only needs 1
interrupt every few seconds for freeing resources.
Bruce
_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
