Re: [jkirsher/next-queue PATCH] ixgbe: Update adaptive ITR algorithm

[Jesper Dangaard Brouer]

On Mon, 25 Sep 2017 14:55:36 -0700
Alexander Duyck  wrote:

> From: Alexander Duyck 
> 
> The following change is meant to update the adaptive ITR algorithm to
> better support the needs of the network. Specifically with this change what
> I have done is make it so that our ITR algorithm will try to prevent either
> starving a socket buffer for memory in the case of Tx, or overruing an Rx
> socket buffer on receive.
> 
> In addition a side effect of the calculations used is that we should
> function better with new features such as XDP which can handle small
> packets at high rates without needing to lock us into NAPI polling mode.
> 
> Signed-off-by: Alexander Duyck 
> ---
> 
> So I am putting this out to a wider distribution list than normal for a
> patch like this in order to get feedback on if there are any areas I may
> have overlooked. With this patch is should address many of the performance
> limitations seen with pktgen and XDP in terms of workloads that the old
> adaptive scheme wasn't handling.

Thanks a lot Alex!

I've tested the patch with XDP redirect (map), and the issue I reported
in [1] is solved with this patch.

[1] Subject: "XDP redirect measurements, gotchas and tracepoints"
 http://lkml.kernel.org/r/20170821212506.1cb0d...@redhat.com

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

[jkirsher/next-queue PATCH] ixgbe: Update adaptive ITR algorithm

[Alexander Duyck]

From: Alexander Duyck 

The following change is meant to update the adaptive ITR algorithm to
better support the needs of the network. Specifically with this change what
I have done is make it so that our ITR algorithm will try to prevent either
starving a socket buffer for memory in the case of Tx, or overruing an Rx
socket buffer on receive.

In addition a side effect of the calculations used is that we should
function better with new features such as XDP which can handle small
packets at high rates without needing to lock us into NAPI polling mode.

Signed-off-by: Alexander Duyck 
---

So I am putting this out to a wider distribution list than normal for a
patch like this in order to get feedback on if there are any areas I may
have overlooked. With this patch is should address many of the performance
limitations seen with pktgen and XDP in terms of workloads that the old
adaptive scheme wasn't handling.

 drivers/net/ethernet/intel/ixgbe/ixgbe.h  |7 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c  |   11 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c |  215 +++--
 3 files changed, 178 insertions(+), 55 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h 
b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
index 56039d04b38d..555eb80d8a08 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
@@ -435,8 +435,15 @@ static inline unsigned int ixgbe_rx_pg_order(struct 
ixgbe_ring *ring)
 }
 #define ixgbe_rx_pg_size(_ring) (PAGE_SIZE << ixgbe_rx_pg_order(_ring))
 
+#define IXGBE_ITR_ADAPTIVE_MIN_INC 2
+#define IXGBE_ITR_ADAPTIVE_MIN_USECS   10
+#define IXGBE_ITR_ADAPTIVE_MAX_USECS   126
+#define IXGBE_ITR_ADAPTIVE_LATENCY 0x80
+#define IXGBE_ITR_ADAPTIVE_BULK0x00
+
 struct ixgbe_ring_container {
struct ixgbe_ring *ring;/* pointer to linked list of rings */
+   unsigned long next_update;  /* jiffies value of last update */
unsigned int total_bytes;   /* total bytes processed this int */
unsigned int total_packets; /* total packets processed this int */
u16 work_limit; /* total work allowed per interrupt */
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c 
b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
index f1bfae0c41d0..8e2a957aca18 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c
@@ -806,6 +806,7 @@ static void ixgbe_add_ring(struct ixgbe_ring *ring,
ring->next = head->ring;
head->ring = ring;
head->count++;
+   head->next_update = jiffies + 1;
 }
 
 /**
@@ -879,8 +880,11 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter 
*adapter,
/* initialize work limits */
q_vector->tx.work_limit = adapter->tx_work_limit;
 
-   /* initialize pointer to rings */
-   ring = q_vector->ring;
+   /* Initialize setting for adaptive ITR */
+   q_vector->tx.itr = IXGBE_ITR_ADAPTIVE_MAX_USECS |
+  IXGBE_ITR_ADAPTIVE_LATENCY;
+   q_vector->rx.itr = IXGBE_ITR_ADAPTIVE_MAX_USECS |
+  IXGBE_ITR_ADAPTIVE_LATENCY;
 
/* intialize ITR */
if (txr_count && !rxr_count) {
@@ -897,6 +901,9 @@ static int ixgbe_alloc_q_vector(struct ixgbe_adapter 
*adapter,
q_vector->itr = adapter->rx_itr_setting;
}
 
+   /* initialize pointer to rings */
+   ring = q_vector->ring;
+
while (txr_count) {
/* assign generic ring traits */
ring->dev = >pdev->dev;
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c 
b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 3d3739f103af..44a96878075b 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -2517,50 +2517,174 @@ enum latency_range {
 static void ixgbe_update_itr(struct ixgbe_q_vector *q_vector,
 struct ixgbe_ring_container *ring_container)
 {
-   int bytes = ring_container->total_bytes;
-   int packets = ring_container->total_packets;
-   u32 timepassed_us;
-   u64 bytes_perint;
-   u8 itr_setting = ring_container->itr;
+   unsigned int itr = IXGBE_ITR_ADAPTIVE_MIN_USECS |
+  IXGBE_ITR_ADAPTIVE_LATENCY;
+   unsigned int avg_wire_size, packets, bytes;
+   unsigned long next_update = jiffies;
 
-   if (packets == 0)
+   /* If we don't have any rings just leave ourselves set for maximum
+* possible latency so we take ourselves out of the equation.
+*/
+   if (!ring_container->ring)
return;
 
-   /* simple throttlerate management
-*   0-10MB/s   lowest (10 ints/s)
-*  10-20MB/s   low(2 ints/s)
-*  20-1249MB/s bulk   (12000 ints/s)
+   /* If we didn't update within