Re: mbuf_jumbo_9k & iSCSI failing

[Hans Petter Selasky]

On 09/22/17 22:33, Ben RUBSON wrote:

On 22 Sep 2017, at 20:48, Ryan Stone  wrote:

Hans and I have proposed different approaches to the problem.  I was
taken off this issue at $WORK for a while, but coincidentally I just
picked it up again in the last week or so.  I'm working on evaluating
the performance characteristics of the two approaches and once I'm
satisfied with that I'll work with Hans to get a solution into the
tree.


Many thanks for the update Ryan !
Good timing again on the email :)



Hi Ryan,

If I didn't send you patches for testing for JUMBO 9K and acceleration 
of small packet RX - ping me off-list.


--HPS
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Ben RUBSON]

> On 22 Sep 2017, at 20:48, Ryan Stone  wrote:
> 
> Hans and I have proposed different approaches to the problem.  I was
> taken off this issue at $WORK for a while, but coincidentally I just
> picked it up again in the last week or so.  I'm working on evaluating
> the performance characteristics of the two approaches and once I'm
> satisfied with that I'll work with Hans to get a solution into the
> tree.

Many thanks for the update Ryan !
Good timing again on the email :)

Ben
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Ryan Stone]

Hans and I have proposed different approaches to the problem.  I was
taken off this issue at $WORK for a while, but coincidentally I just
picked it up again in the last week or so.  I'm working on evaluating
the performance characteristics of the two approaches and once I'm
satisfied with that I'll work with Hans to get a solution into the
tree.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Ben RUBSON]

> On 12 Jul 2017, at 01:02, Ryan Stone  wrote:
> 
> I've just put up a review that fixes mlx4_en to no longer use clusters larger 
> than PAGE_SIZE in its receive path.  The patch is based off of the older 
> version of the driver which did the same, but keeps all of the changes to the 
> driver since then (including support for bus_dma).  The review can be found 
> here:
> 
> https://reviews.freebsd.org/D11560 

Hi Ryan, Hans, Matt... and lists :)
May I ask you some news, did you manage to find a suitable way to solve this 
issue ?
Many thanks !
Ben
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Ryan Stone]

I've just put up a review that fixes mlx4_en to no longer use clusters
larger than PAGE_SIZE in its receive path.  The patch is based off of the
older version of the driver which did the same, but keeps all of the
changes to the driver since then (including support for bus_dma).  The
review can be found here:

https://reviews.freebsd.org/D11560
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Zaphod Beeblebrox]

Don't forget that, generally, as I understand it, the network stack suffers
from the same problem for 9k buffers.

On Sun, Jun 25, 2017 at 12:56 PM, Ben RUBSON  wrote:

> > On 25 Jun 2017, at 17:32, Ryan Stone  wrote:
> >
> > Having looking at the original email more closely, I see that you showed
> an mlxen interface with a 9020 MTU.  Seeing allocation failures of 9k mbuf
> clusters increase while you are far below the zone's limit means that
> you're definitely running into the bug I'm describing, and this bug could
> plausibly cause the iSCSI errors that you describe.
> >
> > The issue is that the newer version of the driver tries to allocate a
> single buffer to accommodate an MTU-sized packet.  Over time, however,
> memory will become fragmented and eventually it can become impossible to
> allocate a 9k physically contiguous buffer.  When this happens the driver
> is unable to allocate buffers to receive packets and is forced to drop
> them.  Presumably, if iSCSI suffers too many packet drops it will terminate
> the connection.  The older version of the driver limited itself to
> page-sized buffers, so it was immune to issues with memory fragmentation.
>
> Thank you for your explanation Ryan.
> You say "over time", and you're right, I have to wait several days (here
> 88) before the problem occurs.
> Strange however that in 2500MB free memory system is unable to find 9k
> physically contiguous. But we never know :)
>
> Let's then wait for your patch !
> (and reboot for now)
>
> Many thx !
>
> Ben
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Andrey V. Elsukov]

On 26.06.2017 19:26, Matt Joras wrote:
> I didn't think that ixgbe(4) still suffered from this problem, and we
> use it in the same situations rstone mentioned above. Indeed, ixgbe(4)
> doesn't presently suffer from this problem (you can see that in your
> patch, as it is only effectively changing the other drivers), though
> it used to. It looks like it was first fixed to not to in r280182.
> 

Yes, actually we have this patch since 8.x. Recent drivers aren't
affected by this problem. iflib also has the code:

#ifndef CONTIGMALLOC_WORKS
else
fl->ifl_buf_size = MJUMPAGESIZE;
#else
else if (sctx->isc_max_frame_size <= 4096)
fl->ifl_buf_size = MJUMPAGESIZE;
else if (sctx->isc_max_frame_size <= 9216)
fl->ifl_buf_size = MJUM9BYTES;
else
fl->ifl_buf_size = MJUM16BYTES;
#endif

that seems by default doesn't use 9-16k mbufs.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: mbuf_jumbo_9k & iSCSI failing

[Matt Joras]

On Mon, Jun 26, 2017 at 6:36 AM, Andrey V. Elsukov  wrote:
> On 26.06.2017 16:29, Ben RUBSON wrote:
>>
>>> On 26 Jun 2017, at 15:25, Andrey V. Elsukov  wrote:
>>>
>>> On 26.06.2017 16:27, Ben RUBSON wrote:

> On 26 Jun 2017, at 15:13, Andrey V. Elsukov  wrote:
>
> I think it is not mlxen specific problem, we have the same symptoms with
> ixgbe(4) driver too. To avoid the problem we have patches that are
> disable using of 9k mbufs, and instead only use 4k mbufs.

 Interesting feedback Andrey, thank you !
 The problem may be then "general".
 So you still use large MTU (>=9000) but only allocating 4k mbufs, as a 
 workaround ?
>>>
>>> Yes.
>>
>> Is it a kernel patch or a driver/ixgbe patch ?
>
> I attached it.
>
> --
> WBR, Andrey V. Elsukov

I didn't think that ixgbe(4) still suffered from this problem, and we
use it in the same situations rstone mentioned above. Indeed, ixgbe(4)
doesn't presently suffer from this problem (you can see that in your
patch, as it is only effectively changing the other drivers), though
it used to. It looks like it was first fixed to not to in r280182.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[YongHyeon PYUN]

On Mon, Jun 26, 2017 at 03:44:58PM +0200, Julien Cigar wrote:
> On Mon, Jun 26, 2017 at 04:13:33PM +0300, Andrey V. Elsukov wrote:
> > On 25.06.2017 18:32, Ryan Stone wrote:
> > > Having looking at the original email more closely, I see that you showed 
> > > an
> > > mlxen interface with a 9020 MTU.  Seeing allocation failures of 9k mbuf
> > > clusters increase while you are far below the zone's limit means that
> > > you're definitely running into the bug I'm describing, and this bug could
> > > plausibly cause the iSCSI errors that you describe.
> > > 
> > > The issue is that the newer version of the driver tries to allocate a
> > > single buffer to accommodate an MTU-sized packet.  Over time, however,
> > > memory will become fragmented and eventually it can become impossible to
> > > allocate a 9k physically contiguous buffer.  When this happens the driver
> > > is unable to allocate buffers to receive packets and is forced to drop
> > > them.  Presumably, if iSCSI suffers too many packet drops it will 
> > > terminate
> > > the connection.  The older version of the driver limited itself to
> > > page-sized buffers, so it was immune to issues with memory fragmentation.
> > 
> > I think it is not mlxen specific problem, we have the same symptoms with
> > ixgbe(4) driver too. To avoid the problem we have patches that are
> > disable using of 9k mbufs, and instead only use 4k mbufs.
> 
> I had the same issue on a lightly loaded HP DL20 machine (BCM5720 
> chipsets), 8GB of RAM, running 10.3. Problem usually happens
> within 30 days with 9k jumbo clusters allocation failure.
> 

This looks strange to me.  If I recall correctly bge(4) does not
request physically contiguous 9k jumbo buffers for BCM5720 so it
wouldn't suffer from memory fragmentation. (It uses m_cljget() and
takes advantage of extended RX BDs to handle up to 4 DMA segments).
If your controller is either BCM5714/BCM5715 or BCM5780, it
requires physically contiguous 9k jumbo buffers to handle jumbo
frames though.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Ben RUBSON]

> On 26 Jun 2017, at 15:13, Andrey V. Elsukov  wrote:
> 
> I think it is not mlxen specific problem, we have the same symptoms with
> ixgbe(4) driver too. To avoid the problem we have patches that are
> disable using of 9k mbufs, and instead only use 4k mbufs.

Another workaround is to decrease the MTU until 9K mbufs are not more used.
On my systems it gives a 4072 bytes MTU.
It solved the issue without having to reboot.
Of course it's just a workaround, as decreasing MTU increases overhead...
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Julien Cigar]

On Mon, Jun 26, 2017 at 04:13:33PM +0300, Andrey V. Elsukov wrote:
> On 25.06.2017 18:32, Ryan Stone wrote:
> > Having looking at the original email more closely, I see that you showed an
> > mlxen interface with a 9020 MTU.  Seeing allocation failures of 9k mbuf
> > clusters increase while you are far below the zone's limit means that
> > you're definitely running into the bug I'm describing, and this bug could
> > plausibly cause the iSCSI errors that you describe.
> > 
> > The issue is that the newer version of the driver tries to allocate a
> > single buffer to accommodate an MTU-sized packet.  Over time, however,
> > memory will become fragmented and eventually it can become impossible to
> > allocate a 9k physically contiguous buffer.  When this happens the driver
> > is unable to allocate buffers to receive packets and is forced to drop
> > them.  Presumably, if iSCSI suffers too many packet drops it will terminate
> > the connection.  The older version of the driver limited itself to
> > page-sized buffers, so it was immune to issues with memory fragmentation.
> 
> I think it is not mlxen specific problem, we have the same symptoms with
> ixgbe(4) driver too. To avoid the problem we have patches that are
> disable using of 9k mbufs, and instead only use 4k mbufs.

I had the same issue on a lightly loaded HP DL20 machine (BCM5720 
chipsets), 8GB of RAM, running 10.3. Problem usually happens
within 30 days with 9k jumbo clusters allocation failure.

> 
> -- 
> WBR, Andrey V. Elsukov
> 




-- 
Julien Cigar
Belgian Biodiversity Platform (http://www.biodiversity.be)
PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.


signature.asc
Description: PGP signature

Re: mbuf_jumbo_9k & iSCSI failing

[Ben RUBSON]

> On 26 Jun 2017, at 15:36, Andrey V. Elsukov  wrote:
> 
> On 26.06.2017 16:29, Ben RUBSON wrote:
>> 
>>> On 26 Jun 2017, at 15:25, Andrey V. Elsukov  wrote:
>>> 
>>> On 26.06.2017 16:27, Ben RUBSON wrote:
 
> On 26 Jun 2017, at 15:13, Andrey V. Elsukov  wrote:
> 
> I think it is not mlxen specific problem, we have the same symptoms with
> ixgbe(4) driver too. To avoid the problem we have patches that are
> disable using of 9k mbufs, and instead only use 4k mbufs.
 
 Interesting feedback Andrey, thank you !
 The problem may be then "general".
 So you still use large MTU (>=9000) but only allocating 4k mbufs, as a 
 workaround ?
>>> 
>>> Yes.
>> 
>> Is it a kernel patch or a driver/ixgbe patch ?
> 
> I attached it.

Thank you !
The idea of new sysctls to enable/disable the workaround is nice.
Should be easy to modify to use with mlx4_en, waiting for Ryan specific work on 
this driver. 

I found a similar issue, reported date : 2013-10-28
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=183381

FreeBSD certainly needs a general solid patch !
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Andrey V. Elsukov]

On 26.06.2017 16:29, Ben RUBSON wrote:
> 
>> On 26 Jun 2017, at 15:25, Andrey V. Elsukov  wrote:
>>
>> On 26.06.2017 16:27, Ben RUBSON wrote:
>>>
 On 26 Jun 2017, at 15:13, Andrey V. Elsukov  wrote:

 I think it is not mlxen specific problem, we have the same symptoms with
 ixgbe(4) driver too. To avoid the problem we have patches that are
 disable using of 9k mbufs, and instead only use 4k mbufs.
>>>
>>> Interesting feedback Andrey, thank you !
>>> The problem may be then "general".
>>> So you still use large MTU (>=9000) but only allocating 4k mbufs, as a 
>>> workaround ?
>>
>> Yes.
> 
> Is it a kernel patch or a driver/ixgbe patch ?

I attached it.

-- 
WBR, Andrey V. Elsukov
From 57b4789b7f6699a581ded2f4e07c7b12737af1e7 Mon Sep 17 00:00:00 2001
From: "Andrey V. Elsukov" 
Date: Thu, 6 Oct 2016 14:56:37 +0300
Subject: [PATCH 04/65] Add m_preferredsize() and use it in all intel's
 drivers.

---
 sys/dev/e1000/if_em.c |  7 +--
 sys/dev/e1000/if_igb.c|  7 +--
 sys/dev/ixgbe/if_ix.c |  5 +
 sys/dev/ixgbe/if_ixv.c|  5 +
 sys/dev/ixl/if_ixlv.c |  5 +
 sys/dev/ixl/ixl_pf_main.c |  5 +
 sys/kern/kern_mbuf.c  | 35 +++
 sys/sys/mbuf.h|  1 +
 8 files changed, 42 insertions(+), 28 deletions(-)

diff --git a/sys/dev/e1000/if_em.c b/sys/dev/e1000/if_em.c
index 7e2690eae08..1af66b7c519 100644
--- a/sys/dev/e1000/if_em.c
+++ b/sys/dev/e1000/if_em.c
@@ -1421,12 +1421,7 @@ em_init_locked(struct adapter *adapter)
 	** Figure out the desired mbuf
 	** pool for doing jumbos
 	*/
-	if (adapter->hw.mac.max_frame_size <= 2048)
-		adapter->rx_mbuf_sz = MCLBYTES;
-	else if (adapter->hw.mac.max_frame_size <= 4096)
-		adapter->rx_mbuf_sz = MJUMPAGESIZE;
-	else
-		adapter->rx_mbuf_sz = MJUM9BYTES;
+	adapter->rx_mbuf_sz = m_preferredsize(adapter->hw.mac.max_frame_size);
 
 	/* Prepare receive descriptors and buffers */
 	if (em_setup_receive_structures(adapter)) {
diff --git a/sys/dev/e1000/if_igb.c b/sys/dev/e1000/if_igb.c
index 8e018995029..bfaecae1f71 100644
--- a/sys/dev/e1000/if_igb.c
+++ b/sys/dev/e1000/if_igb.c
@@ -1325,12 +1325,7 @@ igb_init_locked(struct adapter *adapter)
 	** Figure out the desired mbuf pool
 	** for doing jumbo/packetsplit
 	*/
-	if (adapter->max_frame_size <= 2048)
-		adapter->rx_mbuf_sz = MCLBYTES;
-	else if (adapter->max_frame_size <= 4096)
-		adapter->rx_mbuf_sz = MJUMPAGESIZE;
-	else
-		adapter->rx_mbuf_sz = MJUM9BYTES;
+	adapter->rx_mbuf_sz = m_preferredsize(adapter->max_frame_size);
 
 	/* Prepare receive descriptors and buffers */
 	if (igb_setup_receive_structures(adapter)) {
diff --git a/sys/dev/ixgbe/if_ix.c b/sys/dev/ixgbe/if_ix.c
index cf2231dc8fc..26fce2704ba 100644
--- a/sys/dev/ixgbe/if_ix.c
+++ b/sys/dev/ixgbe/if_ix.c
@@ -1118,10 +1118,7 @@ ixgbe_init_locked(struct adapter *adapter)
 	ixgbe_set_multi(adapter);
 
 	/* Determine the correct mbuf pool, based on frame size */
-	if (adapter->max_frame_size <= MCLBYTES)
-		adapter->rx_mbuf_sz = MCLBYTES;
-	else
-		adapter->rx_mbuf_sz = MJUMPAGESIZE;
+	adapter->rx_mbuf_sz = m_preferredsize(adapter->max_frame_size);
 
 	/* Prepare receive descriptors and buffers */
 	if (ixgbe_setup_receive_structures(adapter)) {
diff --git a/sys/dev/ixgbe/if_ixv.c b/sys/dev/ixgbe/if_ixv.c
index 80fb1b34be3..5062affb779 100644
--- a/sys/dev/ixgbe/if_ixv.c
+++ b/sys/dev/ixgbe/if_ixv.c
@@ -698,10 +698,7 @@ ixv_init_locked(struct adapter *adapter)
 	** Determine the correct mbuf pool
 	** for doing jumbo/headersplit
 	*/
-	if (ifp->if_mtu > ETHERMTU)
-		adapter->rx_mbuf_sz = MJUMPAGESIZE;
-	else
-		adapter->rx_mbuf_sz = MCLBYTES;
+	adapter->rx_mbuf_sz = m_preferredsize(ifp->if_mtu);
 
 	/* Prepare receive descriptors and buffers */
 	if (ixgbe_setup_receive_structures(adapter)) {
diff --git a/sys/dev/ixl/if_ixlv.c b/sys/dev/ixl/if_ixlv.c
index c447c34689e..608d784bfee 100644
--- a/sys/dev/ixl/if_ixlv.c
+++ b/sys/dev/ixl/if_ixlv.c
@@ -904,10 +904,7 @@ ixlv_init_locked(struct ixlv_sc *sc)
 
 		ixl_init_tx_ring(que);
 
-		if (vsi->max_frame_size <= MCLBYTES)
-			rxr->mbuf_sz = MCLBYTES;
-		else
-			rxr->mbuf_sz = MJUMPAGESIZE;
+		rxr->mbuf_sz = m_preferredsize(vsi->max_frame_size);
 		ixl_init_rx_ring(que);
 	}
 
diff --git a/sys/dev/ixl/ixl_pf_main.c b/sys/dev/ixl/ixl_pf_main.c
index d8da4cfee10..8600b0f931e 100644
--- a/sys/dev/ixl/ixl_pf_main.c
+++ b/sys/dev/ixl/ixl_pf_main.c
@@ -2067,10 +2067,7 @@ ixl_initialize_vsi(struct ixl_vsi *vsi)
 		ixl_init_tx_ring(que);
 
 		/* Next setup the HMC RX Context  */
-		if (vsi->max_frame_size <= MCLBYTES)
-			rxr->mbuf_sz = MCLBYTES;
-		else
-			rxr->mbuf_sz = MJUMPAGESIZE;
+		rxr->mbuf_sz = m_preferredsize(vsi->max_frame_size);
 
 		u16 max_rxmax = rxr->mbuf_sz * hw->func_caps.rx_buf_chain_len;
 
diff --git a/sys/kern/kern_mbuf.c b/sys/kern/kern_mbuf.c
index 0d0c1c86b16..7c10cedb075 100644
--- a/sys/kern/kern_mbuf.c
+++ b/sys/kern/kern_mbuf.c
@@ -103,6 +103,10 

Re: mbuf_jumbo_9k & iSCSI failing

[Ben RUBSON]

> On 26 Jun 2017, at 15:25, Andrey V. Elsukov  wrote:
> 
> On 26.06.2017 16:27, Ben RUBSON wrote:
>> 
>>> On 26 Jun 2017, at 15:13, Andrey V. Elsukov  wrote:
>>> 
>>> I think it is not mlxen specific problem, we have the same symptoms with
>>> ixgbe(4) driver too. To avoid the problem we have patches that are
>>> disable using of 9k mbufs, and instead only use 4k mbufs.
>> 
>> Interesting feedback Andrey, thank you !
>> The problem may be then "general".
>> So you still use large MTU (>=9000) but only allocating 4k mbufs, as a 
>> workaround ?
> 
> Yes.

Is it a kernel patch or a driver/ixgbe patch ?
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Andrey V. Elsukov]

On 26.06.2017 16:27, Ben RUBSON wrote:
> 
>> On 26 Jun 2017, at 15:13, Andrey V. Elsukov  wrote:
>>
>> I think it is not mlxen specific problem, we have the same symptoms with
>> ixgbe(4) driver too. To avoid the problem we have patches that are
>> disable using of 9k mbufs, and instead only use 4k mbufs.
> 
> Interesting feedback Andrey, thank you !
> The problem may be then "general".
> So you still use large MTU (>=9000) but only allocating 4k mbufs, as a 
> workaround ?

Yes.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: mbuf_jumbo_9k & iSCSI failing

[Ben RUBSON]

> On 26 Jun 2017, at 15:13, Andrey V. Elsukov  wrote:
> 
> I think it is not mlxen specific problem, we have the same symptoms with
> ixgbe(4) driver too. To avoid the problem we have patches that are
> disable using of 9k mbufs, and instead only use 4k mbufs.

Interesting feedback Andrey, thank you !
The problem may be then "general".
So you still use large MTU (>=9000) but only allocating 4k mbufs, as a 
workaround ?
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Andrey V. Elsukov]

On 25.06.2017 18:32, Ryan Stone wrote:
> Having looking at the original email more closely, I see that you showed an
> mlxen interface with a 9020 MTU.  Seeing allocation failures of 9k mbuf
> clusters increase while you are far below the zone's limit means that
> you're definitely running into the bug I'm describing, and this bug could
> plausibly cause the iSCSI errors that you describe.
> 
> The issue is that the newer version of the driver tries to allocate a
> single buffer to accommodate an MTU-sized packet.  Over time, however,
> memory will become fragmented and eventually it can become impossible to
> allocate a 9k physically contiguous buffer.  When this happens the driver
> is unable to allocate buffers to receive packets and is forced to drop
> them.  Presumably, if iSCSI suffers too many packet drops it will terminate
> the connection.  The older version of the driver limited itself to
> page-sized buffers, so it was immune to issues with memory fragmentation.

I think it is not mlxen specific problem, we have the same symptoms with
ixgbe(4) driver too. To avoid the problem we have patches that are
disable using of 9k mbufs, and instead only use 4k mbufs.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: mbuf_jumbo_9k & iSCSI failing

[Edward Napierala]

2017-06-25 16:32 GMT+01:00 Ryan Stone :

> Having looking at the original email more closely, I see that you showed an
> mlxen interface with a 9020 MTU.  Seeing allocation failures of 9k mbuf
> clusters increase while you are far below the zone's limit means that
> you're definitely running into the bug I'm describing, and this bug could
> plausibly cause the iSCSI errors that you describe.
>
> The issue is that the newer version of the driver tries to allocate a
> single buffer to accommodate an MTU-sized packet.  Over time, however,
> memory will become fragmented and eventually it can become impossible to
> allocate a 9k physically contiguous buffer.  When this happens the driver
> is unable to allocate buffers to receive packets and is forced to drop
> them.  Presumably, if iSCSI suffers too many packet drops it will terminate
> the connection.  [..]


More specifically, it will terminate the connection when there's no "ping
reply"
from the other side for the configured amount of time, which defaults to
five
seconds.  It can be changed using the kern.iscsi.ping_timeout sysctl, as
described in iscsi(4).
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Ben RUBSON]

> On 25 Jun 2017, at 17:32, Ryan Stone  wrote:
> 
> Having looking at the original email more closely, I see that you showed an 
> mlxen interface with a 9020 MTU.  Seeing allocation failures of 9k mbuf 
> clusters increase while you are far below the zone's limit means that you're 
> definitely running into the bug I'm describing, and this bug could plausibly 
> cause the iSCSI errors that you describe.
> 
> The issue is that the newer version of the driver tries to allocate a single 
> buffer to accommodate an MTU-sized packet.  Over time, however, memory will 
> become fragmented and eventually it can become impossible to allocate a 9k 
> physically contiguous buffer.  When this happens the driver is unable to 
> allocate buffers to receive packets and is forced to drop them.  Presumably, 
> if iSCSI suffers too many packet drops it will terminate the connection.  The 
> older version of the driver limited itself to page-sized buffers, so it was 
> immune to issues with memory fragmentation.

Thank you for your explanation Ryan.
You say "over time", and you're right, I have to wait several days (here 88) 
before the problem occurs.
Strange however that in 2500MB free memory system is unable to find 9k 
physically contiguous. But we never know :)

Let's then wait for your patch !
(and reboot for now)

Many thx !

Ben
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Ryan Stone]

Having looking at the original email more closely, I see that you showed an
mlxen interface with a 9020 MTU.  Seeing allocation failures of 9k mbuf
clusters increase while you are far below the zone's limit means that
you're definitely running into the bug I'm describing, and this bug could
plausibly cause the iSCSI errors that you describe.

The issue is that the newer version of the driver tries to allocate a
single buffer to accommodate an MTU-sized packet.  Over time, however,
memory will become fragmented and eventually it can become impossible to
allocate a 9k physically contiguous buffer.  When this happens the driver
is unable to allocate buffers to receive packets and is forced to drop
them.  Presumably, if iSCSI suffers too many packet drops it will terminate
the connection.  The older version of the driver limited itself to
page-sized buffers, so it was immune to issues with memory fragmentation.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Ben RUBSON]

> On 25 Jun 2017, at 17:14, Ryan Stone  wrote:
> 
> Is this setup using the mlx4_en driver?  If so, recent versions of that 
> driver has a regression when using MTUs greater than the page size (4096 on 
> i386/amd64).  The bug will cause the card to drop packets when the system is 
> under memory pressure, and in certain causes the card can get into a state 
> when it is no longer able to receive packets.  I am working on a fix; I can 
> post a patch when it's complete.

Thank you very much for your feedback Ryan.

Yes, my system is using mlx4_en driver, the one directly from FreeBSD 11.0 
sources tree.
Any indicator I could catch to be sure I'm experiencing the issue you are 
working on ?

Sounds like anyway I may be suffering from it...
Of course I would be glad to help testing your patch when it's complete.

Thank you again,

Ben

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: mbuf_jumbo_9k & iSCSI failing

[Ryan Stone]

Is this setup using the mlx4_en driver?  If so, recent versions of that
driver has a regression when using MTUs greater than the page size (4096 on
i386/amd64).  The bug will cause the card to drop packets when the system
is under memory pressure, and in certain causes the card can get into a
state when it is no longer able to receive packets.  I am working on a fix;
I can post a patch when it's complete.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

mbuf_jumbo_9k & iSCSI failing

[Ben RUBSON]

> On 30 Dec 2016, at 22:55, Ben RUBSON  wrote:
> 
> Hello,
> 
> 2 FreeBSD 11.0-p3 servers, one iSCSI initiator, one target.
> Both with Mellanox ConnectX-3 40G.
> 
> Since a few days, sometimes, under undetermined circumstances, as soon as 
> there is some (very low) iSCSI traffic, some of the disks get disconnected :
> kernel: WARNING: 192.168.2.2 (iqn..): no ping reply (NOP-Out) after 5 
> seconds; dropping connection
> 
> At the same moment, sysctl counters hw.mlxen1.stat.rx_ring*.error grow on 
> initiator side.
> 
> I then tried to reproduce these network errors burning the link at 40G 
> full-duplex using iPerf.
> But I did not manage to increase these error counters.
> 
> It's strange because it's a sporadic issue, I can have traffic on iSCSI disks 
> without any issue, and sometimes, they get disconnected with errors growing.

> On 01 Jan 2017, at 09:16, Meny Yossefi  wrote:
> 
> Any chance you ran out of mbufs in the system?

> On 02 Jan 2017, at 12:09, Ben RUBSON  wrote:
> 
> I think you are right, this could be a mbufs issue.
> Here are some more numbers :
> 
> # vmstat -z | grep -v "0,   0$"
> ITEM   SIZE   LIMIT USED FREE REQ  FAIL 
> SLEEP
> 4 Bucket:32,  0,2673,   28327,   88449799,17317, 0
> 8 Bucket:64,  0, 449,   15609,   13926386, 4871, 0
> 12 Bucket:   96,  0, 335,5323,   10293892,   142872, 0
> 16 Bucket:  128,  0, 533,6070,7618615,   472647, 0
> 32 Bucket:  256,  0,8317,   22133,   36020376,   563479, 0
> 64 Bucket:  512,  0,1238,3298,   20138111, 11430742, 0
> 128 Bucket:1024,  0,1865,2963,   21162182,   158752, 0
> 256 Bucket:2048,  0,1626, 450,   80253784,  4890164, 0
> mbuf_jumbo_9k: 9216, 603712,   16400,8744, 4128521064, 2661, 0

> On 03 Jan 2017, at 07:27, Meny Yossefi  wrote:
> 
> Have you tried increasing the mbufs limit? 
> (sysctl) kern.ipc.nmbufs (Maximum number of mbufs allowed)

> On 04 Jan 2017, at 14:47, Ben RUBSON  wrote:
> 
> No I did not try this yet.
> However, from the numbers above (and below), I think I should increase 
> kern.ipc.nmbjumbo9 instead ?

> On 30 Jan 2017, at 15:36, Ben RUBSON  wrote:
> 
> So, to give some news, increasing kern.ipc.nmbjumbo9 helped a lot.
> Just a very little issue (compared to the others before) over the last 3 
> weeks.



Hello,

I'm back today with this issue.
Above is my discussion with Meny from Mellanox at the beginning of 2017.
(topic was "iSCSI failing, MLX rx_ring errors ?", on freebsd-net list)

So this morning issue came again, some of my iSCSI disks were disconnected.
Below are some numbers.



# vmstat -z | grep -v "0,   0$"
ITEM   SIZELIMIT USED FREE REQ FAIL SLEEP
8 Bucket:64,   0, 654,8522,   28604967,  11, 0
12 Bucket:   96,   0, 976,5092,   23758734,  78, 0
32 Bucket:  256,   0, 789,4491,   43446969, 137, 0
64 Bucket:  512,   0, 666,2750,   47568959, 1272018, 0
128 Bucket:1024,   0,1047,1249,   28774042,  232504, 0
256 Bucket:2048,   0,1611, 369,  139988097, 8931139, 0
vmem btag:   56,   0, 2949738,   15506,   18092235,   20908, 0
mbuf_jumbo_9k: 9216, 2037529,   16400,8776, 8610737115, 297, 0

# uname -rs
FreeBSD 11.0-RELEASE-p8

# uptime
 3:34p.m.  up 88 days, 15:57, 2 users, load averages: 0.95, 0.67, 0.62

# grep kern.ipc.nmb /boot/loader.conf 
kern.ipc.nmbjumbo9=2037529
kern.ipc.nmbjumbo16=1

# sysctl kern.ipc | grep mb
kern.ipc.nmbufs: 26080380
kern.ipc.nmbjumbo16: 4
kern.ipc.nmbjumbo9: 6112587
kern.ipc.nmbjumbop: 2037529
kern.ipc.nmbclusters: 4075060
kern.ipc.maxmbufmem: 33382887424

# ifconfig mlxen1
mlxen1: flags=8843 metric 0 mtu 9020
options=ed07bb
nd6 options=29
media: Ethernet autoselect (40Gbase-CR4 )
status: active



I just caught the issue growing :

# vmstat -z | grep mbuf_jumbo_9k
ITEM   SIZE  LIMIT USED  FREEREQ FAIL SLEEP
mbuf_jumbo_9k: 9216, 2037529, 16415, 7316,8735246407, 665, 0
mbuf_jumbo_9k: 9216, 2037529, 16411, 7320,8735286748, 665, 0
mbuf_jumbo_9k: 9216, 2037529, 16415, 7316,8735298937, 667, 0
mbuf_jumbo_9k: 9216, 2037529, 16438, 7293,8735337634, 667, 0
mbuf_jumbo_9k: 9216, 2037529, 16407, 7324,8735354339, 668, 0
mbuf_jumbo_9k: 9216, 2037529, 16400, 7331,8735382105, 669, 0
mbuf_jumbo_9k: 9216, 2037529, 16402, 7329,8735392836, 671, 0
mbuf_jumbo_9k: 9216, 2037529, 16400, 7331,8735423910, 671, 0