On Fri, Aug 05, 2016 at 11:56:15AM +1000, Darren Tucker wrote:
> On Thu, Aug 04, 2016 at 02:46:44PM +0200, Momtchil Momtchev wrote:
> [...]
> > What is the problem with software interrupt moderation? That it has a
> > fixed timer while the hardware one scales with the RX rate?
>
> The hardware moderation can do per-N-packets in addition to a timer.
>
> > This shouldn't
> > halve the performance? It should be more like 10% to 15% and some latency
> > benefit? I have also noticed that the TX rate is higher than the RX rate
> > (about 320 Mbit/s vs 260 Mbit/s). Could it be that the FreeBSD driver uses
> > MSI interrupts and the OpenBSD one does not?
>
> Dunno. If I knew what the cause was I'd have fixed it :-(
Hey, I might have found it. From my other diff:
> + * According to the Linux driver, supposedly:
> + * (TxTimer << 12) | (TxPackets << 8) | (RxTimer << 4) | RxPackets
however in the header the RXTIME/TXTIME macros didn't match that:
> #define RL_IM_RXTIME(t) ((t) & 0xf)
> +#define RL_IM_RXPKTS(t) (((t) & 0xf) << 4)
> #define RL_IM_TXTIME(t) (((t) & 0xf) << 8)
> +#define RL_IM_TXPKTS(t) (((t) & 0xf) << 12)
so assuming the comment was correct, I wasn't actually setting the holdoff
timers :-(
A quick test with this diff (just routing through it, no PF, no pool
debug) gives me:
$ iperf -c host -i 10 -t 60
------------------------------------------------------------
Client connecting to nfs, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.32.1 port 43092 connected with 192.168.33.44 port
5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 803 MBytes 674 Mbits/sec
[ 3] 10.0-20.0 sec 844 MBytes 708 Mbits/sec
[ 3] 20.0-30.0 sec 876 MBytes 735 Mbits/sec
[ 3] 30.0-40.0 sec 915 MBytes 768 Mbits/sec
[ 3] 40.0-50.0 sec 929 MBytes 779 Mbits/sec
[ 3] 50.0-60.0 sec 917 MBytes 769 Mbits/sec
[ 3] 0.0-60.0 sec 5.16 GBytes 739 Mbits/sec
Index: dev/ic/re.c
===================================================================
RCS file: /cvs/src/sys/dev/ic/re.c,v
retrieving revision 1.192
diff -u -p -r1.192 re.c
--- dev/ic/re.c 20 Apr 2016 12:15:24 -0000 1.192
+++ dev/ic/re.c 9 Aug 2016 00:52:45 -0000
@@ -747,7 +747,7 @@ re_attach(struct rl_softc *sc, const cha
sc->rl_flags |= RL_FLAG_PHYWAKE | RL_FLAG_PHYWAKE_PM |
RL_FLAG_PAR | RL_FLAG_DESCV2 | RL_FLAG_MACSTAT |
RL_FLAG_CMDSTOP | RL_FLAG_AUTOPAD | RL_FLAG_JUMBOV2 |
- RL_FLAG_WOL_MANLINK;
+ RL_FLAG_WOL_MANLINK | RL_FLAG_HWIM;
sc->rl_max_mtu = RL_JUMBO_MTU_9K;
break;
case RL_HWREV_8168E_VL:
@@ -821,13 +821,19 @@ re_attach(struct rl_softc *sc, const cha
/* Reset the adapter. */
re_reset(sc);
- sc->rl_tx_time = 5; /* 125us */
- sc->rl_rx_time = 2; /* 50us */
- if (sc->rl_flags & RL_FLAG_PCIE)
- sc->rl_sim_time = 75; /* 75us */
- else
- sc->rl_sim_time = 125; /* 125us */
- sc->rl_imtype = RL_IMTYPE_SIM; /* simulated interrupt moderation */
+ if (sc->rl_flags & RL_FLAG_HWIM) {
+ /* hardware interrupt moderation */
+ sc->rl_imtype = RL_IMTYPE_HW;
+ sc->rl_tx_time = 5; /* 125us */
+ sc->rl_rx_time = 2; /* 50us */
+ } else {
+ /* simulated interrupt moderation */
+ sc->rl_imtype = RL_IMTYPE_SIM;
+ if (sc->rl_flags & RL_FLAG_PCIE)
+ sc->rl_sim_time = 75; /* 75us */
+ else
+ sc->rl_sim_time = 125; /* 125us */
+ }
if (sc->sc_hwrev == RL_HWREV_8139CPLUS)
sc->rl_bus_speed = 33; /* XXX */
@@ -2233,6 +2239,8 @@ re_stop(struct ifnet *ifp)
void
re_setup_hw_im(struct rl_softc *sc)
{
+ u_int16_t im;
+
KASSERT(sc->rl_flags & RL_FLAG_HWIM);
/*
@@ -2258,11 +2266,15 @@ re_setup_hw_im(struct rl_softc *sc)
* Currently we only know how to set 'timer', but not
* 'number of packets', which should be ~30, as far as I
* tested (sink ~900Kpps, interrupt rate is 30KHz)
- */
- CSR_WRITE_2(sc, RL_IM,
- RL_IM_RXTIME(sc->rl_rx_time) |
- RL_IM_TXTIME(sc->rl_tx_time) |
- RL_IM_MAGIC);
+ *
+ * According to the Linux driver, supposedly:
+ * (TxTimer << 12) | (TxPackets << 8) | (RxTimer << 4) | RxPackets
+ * Linux uses hard coded 0x5151.
+ */
+ im = RL_IM_TXTIME(sc->rl_tx_time) | RL_IM_TXPKTS(15) |
+ RL_IM_RXTIME(sc->rl_rx_time) | RL_IM_RXPKTS(2);
+ printf("setting interrupt moderation %hx\n", im); /* XXX */
+ CSR_WRITE_2(sc, RL_IM, im);
}
void
Index: dev/ic/rtl81x9reg.h
===================================================================
RCS file: /cvs/src/sys/dev/ic/rtl81x9reg.h,v
retrieving revision 1.98
diff -u -p -r1.98 rtl81x9reg.h
--- dev/ic/rtl81x9reg.h 20 Apr 2016 12:15:24 -0000 1.98
+++ dev/ic/rtl81x9reg.h 9 Aug 2016 00:52:45 -0000
@@ -568,9 +568,14 @@
#define RL_RXCFG_CONFIG
(RL_RX_FIFOTHRESH|RL_RX_MAXDMA|RL_RX_BUF_SZ)
#define RL_TXCFG_CONFIG (RL_TXCFG_IFG|RL_TX_MAXDMA)
-#define RL_IM_MAGIC 0x5050
-#define RL_IM_RXTIME(t) ((t) & 0xf)
-#define RL_IM_TXTIME(t) (((t) & 0xf) << 8)
+/*
+ * Interrupt moderation. According to the Linux driver, supposedly:
+ * (TxTimer << 12) | (TxPackets << 8) | (RxTimer << 4) | RxPackets
+ */
+#define RL_IM_RXPKTS(t) ((t) & 0xf)
+#define RL_IM_RXTIME(t) (((t) & 0xf) << 4)
+#define RL_IM_TXPKTS(t) (((t) & 0xf) << 8)
+#define RL_IM_TXTIME(t) (((t) & 0xf) << 12)
struct rl_chain_data {
u_int16_t cur_rx;
--
Darren Tucker (dtucker at zip.com.au)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new)
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
