Re: macb napi strange behavior
Hi,
I gave it a shot. Can you please take a look?
I don't know how to deal with multiple queues since Zynq 7000 has one
queue per interface.
I get a performance improvement of over 110 Mbps in IP forwarding (680
Mbps vs 570 Mbps) and a massive reduction of interrupts.
Patch below.
From: Nicolae Rosia nicolae.ro...@certsign.ro
Date: Fri, 26 Jun 2015 19:44:50 +0300
Subject: [PATCH] macb: use tx napi
---
drivers/net/ethernet/cadence/macb.c | 37 ++---
1 file changed, 34 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/cadence/macb.c
b/drivers/net/ethernet/cadence/macb.c
index 30eedb1..2a3ecf7 100644
--- a/drivers/net/ethernet/cadence/macb.c
+++ b/drivers/net/ethernet/cadence/macb.c
@@ -602,10 +602,13 @@ static void macb_tx_interrupt(struct macb_queue *queue)
{
unsigned int tail;
unsigned int head;
+#if 0
u32 status;
+#endif
struct macb *bp = queue-bp;
u16 queue_index = queue - bp-queues;
+#if 0
status = macb_readl(bp, TSR);
macb_writel(bp, TSR, status);
@@ -614,6 +617,7 @@ static void macb_tx_interrupt(struct macb_queue *queue)
netdev_vdbg(bp-dev, macb_tx_interrupt status = 0x%03lx\n,
(unsigned long)status);
+#endif
head = queue-tx_head;
for (tail = queue-tx_tail; tail != head; tail++) {
@@ -951,12 +955,17 @@ static int macb_poll(struct napi_struct *napi, int budget)
status = macb_readl(bp, RSR);
macb_writel(bp, RSR, status);
-work_done = 0;
+status = macb_readl(bp, TSR);
+if (status) {
+macb_writel(bp, TSR, status);
+macb_tx_interrupt(bp-queues[0]);
+}
netdev_vdbg(bp-dev, poll: status = %08lx, budget = %d\n,
(unsigned long)status, budget);
work_done = bp-macbgem_ops.mog_rx(bp, budget);
+
if (work_done budget) {
napi_complete(napi);
@@ -969,6 +978,16 @@ static int macb_poll(struct napi_struct *napi, int budget)
} else {
macb_writel(bp, IER, MACB_RX_INT_FLAGS);
}
+
+/* Packets received while interrupts were disabled */
+status = macb_readl(bp, TSR);
+if (status) {
+if (bp-caps MACB_CAPS_ISR_CLEAR_ON_WRITE)
+macb_writel(bp, ISR, MACB_BIT(TCOMP));
+napi_reschedule(napi);
+} else {
+macb_writel(bp, IER, MACB_TX_INT_FLAGS);
+}
}
/* TODO: Handle errors */
@@ -1029,8 +1048,17 @@ static irqreturn_t macb_interrupt(int irq, void *dev_id)
break;
}
-if (status MACB_BIT(TCOMP))
-macb_tx_interrupt(queue);
+if (status MACB_BIT(TCOMP)) {
+/* disable tx interrupts */
+queue_writel(queue, IDR, MACB_TX_INT_FLAGS);
+if (bp-caps MACB_CAPS_ISR_CLEAR_ON_WRITE)
+queue_writel(queue, ISR, MACB_BIT(TCOMP));
+
+if (napi_schedule_prep(bp-napi)) {
+netdev_vdbg(bp-dev, scheduling TX softirq\n);
+__napi_schedule(bp-napi);
+}
+}
/*
* Link change detection isn't possible with RMII, so we'll
@@ -2348,6 +2376,7 @@ static int macb_probe(struct platform_device *pdev)
}
macb_probe_queues(mem, queue_mask, num_queues);
+dev_err(pdev-dev, queue count: %d.\n, num_queues);
dev = alloc_etherdev_mq(sizeof(*bp), num_queues);
if (!dev)
goto err_out_disable_clocks;
@@ -2377,6 +2406,8 @@ static int macb_probe(struct platform_device *pdev)
if (!(queue_mask (1 hw_q)))
continue;
+dev_err(pdev-dev, queue index: %d\n, q);
+
queue = bp-queues[q];
queue-bp = bp;
if (hw_q) {
--
2.1.0
On Sat, Jun 20, 2015 at 7:43 PM, Francois Romieu rom...@fr.zoreil.com wrote:
Florian Fainelli f.faine...@gmail.com :
[...]
Typically, NAPI is used at the receive side of the Ethernet NIC/driver
to lower the hard/soft interrupt context switch, although there is
nothing that prevent you to implement a similar scheme for the
transmit side. Usually, for transmit you will be submitting one packet
for transmission and get a completion interrupt, so without interrupt
coalescing (software or hardware) you can end-up with 1 interrupt per
packet transmitted.
The wording is a bit shy: there is a long standing policy to move
everything to NAPI context (as well as go mostly lockless, etc.).
Any taker to move macb Tx processing to NAPI context or should I consider it ?
--
Ueimor
--
To unsubscribe from this list: send the line unsubscribe netdev in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: macb napi strange behavior
Nicolae Rosia nicolae.ro...@gmail.com : [...] I gave it a shot. Can you please take a look? I don't know how to deal with multiple queues since Zynq 7000 has one queue per interface. macb_interrupt knows the queue. macb_poll doesn't. Either you store it somewhere or you go for per queue napi struct. -- Ueimor -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: macb napi strange behavior
On Sat, Jun 20, 2015 at 7:43 PM, Francois Romieu rom...@fr.zoreil.com wrote: Florian Fainelli f.faine...@gmail.com : [...] Typically, NAPI is used at the receive side of the Ethernet NIC/driver to lower the hard/soft interrupt context switch, although there is nothing that prevent you to implement a similar scheme for the transmit side. Usually, for transmit you will be submitting one packet for transmission and get a completion interrupt, so without interrupt coalescing (software or hardware) you can end-up with 1 interrupt per packet transmitted. The wording is a bit shy: there is a long standing policy to move everything to NAPI context (as well as go mostly lockless, etc.). Any taker to move macb Tx processing to NAPI context or should I consider it ? I can help with testing! -- To unsubscribe from this list: send the line unsubscribe netdev in
Re: macb napi strange behavior
Florian Fainelli f.faine...@gmail.com : [...] Typically, NAPI is used at the receive side of the Ethernet NIC/driver to lower the hard/soft interrupt context switch, although there is nothing that prevent you to implement a similar scheme for the transmit side. Usually, for transmit you will be submitting one packet for transmission and get a completion interrupt, so without interrupt coalescing (software or hardware) you can end-up with 1 interrupt per packet transmitted. The wording is a bit shy: there is a long standing policy to move everything to NAPI context (as well as go mostly lockless, etc.). Any taker to move macb Tx processing to NAPI context or should I consider it ? -- Ueimor -- To unsubscribe from this list: send the line unsubscribe netdev in
Re: macb napi strange behavior
On 06/17/2015 11:09 AM, Nicolae Rosia wrote: I'm trying to determine why I have a huge number of IRQs for only a macb interface and the other one works just fine (low IRQ activity). I have activated IP forward and I'm just forwarding packets from eth0 to eth1. The platform is Zynq7, Linux kernel 4.0, vanilla macb. cat /proc/interrupts: [...] 144: 679425 0 GIC 54 eth0 145: 17867097 0 GIC 77 eth1 [...] Any ideas? The times we've seen tons of interrupts on Ethernet with interrupts routed through the PL was when the FPGA was unprogrammed (or in the process of being reprogrammed), or was configured with the interrupt line tied to asserted. In the latter case, Linux would eventually stop handling any more interrupts for that port due to the interrupt storm. In the former case, there isn't much one can do except make sure that any FPGA-routed interrupts are unregistered and disabled before FPGA reprogramming and then to re-enable those interrupts after reprogramming. It'd be nice to have some sort of notification to drivers, given when the FPGA state changes, when hardware the drivers are responsible for disappears or gets disconnected. This is an area of research for us at NI. Cheers, Jaeden -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: macb napi strange behavior
2015-06-17 12:00 GMT-07:00 Nicolae Rosia nicolae.ro...@gmail.com: Hi, On Wed, Jun 17, 2015 at 9:54 PM, Jaeden Amero jaeden.am...@ni.com wrote: On 06/17/2015 11:09 AM, Nicolae Rosia wrote: The times we've seen tons of interrupts on Ethernet with interrupts routed through the PL was when the FPGA was unprogrammed (or in the process of being reprogrammed), or was configured with the interrupt line tied to asserted. In the latter case, Linux would eventually stop handling any more interrupts for that port due to the interrupt storm. This isn't the case. The FPGA is programmed, and indeed I'm using the second MAC routed through PL to SFP. The interesting thing is that I'm seeing the exact behavior on the other side (another Zynq7 board), with eth0 having lots of interrupts. It seems that the interface receiving packets doesn't have a high IRQ activity in contrast to the one sending packets. Typically, NAPI is used at the receive side of the Ethernet NIC/driver to lower the hard/soft interrupt context switch, although there is nothing that prevent you to implement a similar scheme for the transmit side. Usually, for transmit you will be submitting one packet for transmission and get a completion interrupt, so without interrupt coalescing (software or hardware) you can end-up with 1 interrupt per packet transmitted. -- Florian -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: macb napi strange behavior
Hi, On Wed, Jun 17, 2015 at 9:54 PM, Jaeden Amero jaeden.am...@ni.com wrote: On 06/17/2015 11:09 AM, Nicolae Rosia wrote: The times we've seen tons of interrupts on Ethernet with interrupts routed through the PL was when the FPGA was unprogrammed (or in the process of being reprogrammed), or was configured with the interrupt line tied to asserted. In the latter case, Linux would eventually stop handling any more interrupts for that port due to the interrupt storm. This isn't the case. The FPGA is programmed, and indeed I'm using the second MAC routed through PL to SFP. The interesting thing is that I'm seeing the exact behavior on the other side (another Zynq7 board), with eth0 having lots of interrupts. It seems that the interface receiving packets doesn't have a high IRQ activity in contrast to the one sending packets. -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
macb napi strange behavior
Hello, I'm trying to determine why I have a huge number of IRQs for only a macb interface and the other one works just fine (low IRQ activity). I have activated IP forward and I'm just forwarding packets from eth0 to eth1. The platform is Zynq7, Linux kernel 4.0, vanilla macb. cat /proc/interrupts: [...] 144: 679425 0 GIC 54 eth0 145: 17867097 0 GIC 77 eth1 [...] Any ideas? Best regards, Nicolae Rosia -- To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
