Re: [pve-devel] [PATCH kernel] Fix #927: add IPoIB performance regression fix

[Michael Rasmussen]

On Tue, 25 Oct 2016 06:16:34 +0200 (CEST)
Dietmar Maurer  wrote:

> > Running LXC over NFS (IPoIB ZFS backed) will very often more or less
> > crash the host when the lxc unmount command runs.  
> 
> And this was stable without the patch?
> 
No, same issue but with the patch performance has gone up. The NFS
issue has been there for all 4.4 kernels.

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael  rasmussen  cc
http://pgp.mit.edu:11371/pks/lookup?op=get=0xD3C9A00E
mir  datanom  net
http://pgp.mit.edu:11371/pks/lookup?op=get=0xE501F51C
mir  miras  org
http://pgp.mit.edu:11371/pks/lookup?op=get=0xE3E80917
--
/usr/games/fortune -es says:
Remember Darwin; building a better mousetrap merely results in smarter
mice.


pgpKW5R5tl_Zz.pgp
Description: OpenPGP digital signature
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH kernel] Fix #927: add IPoIB performance regression fix

[Dietmar Maurer]

> Running LXC over NFS (IPoIB ZFS backed) will very often more or less
> crash the host when the lxc unmount command runs.

And this was stable without the patch?

___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH kernel] Fix #927: add IPoIB performance regression fix

[Michael Rasmussen]

On Tue, 18 Oct 2016 10:46:55 +0200
Wolfgang Bumiller  wrote:

> 
> The 4.4.21-69 kernel package in pvetest should include the fix if you
> want to try it (The last download link in the #927 bugzilla entry should
> still be valid, too, and is a rebuild of the -68 rel)
Has been running several days now. Seems to increase performance and no
problems except this:

Running LXC over NFS (IPoIB ZFS backed) will very often more or less
crash the host when the lxc unmount command runs.

If running the same LXC over NFS (Ethernet ZFS backed) everything works
as expected.

Any ideas?

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael  rasmussen  cc
http://pgp.mit.edu:11371/pks/lookup?op=get=0xD3C9A00E
mir  datanom  net
http://pgp.mit.edu:11371/pks/lookup?op=get=0xE501F51C
mir  miras  org
http://pgp.mit.edu:11371/pks/lookup?op=get=0xE3E80917
--
/usr/games/fortune -es says:
linux: the choice of a GNU generation
(k...@cis.ufl.edu put this on Tshirts in '93)


pgp1xCejSl5QO.pgp
Description: OpenPGP digital signature
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH kernel] Fix #927: add IPoIB performance regression fix

[Wolfgang Bumiller]

On Fri, Oct 14, 2016 at 08:05:56AM +0200, Michael Rasmussen wrote:
> On Thu, 13 Oct 2016 16:49:30 +0200
> Wolfgang Bumiller  wrote:
> 
> > Fixes kernel bug #111921
> > ---
> >  ...ck-the-IB-LL-address-into-the-hard-header.patch | 365 
> > +
> >  Makefile   |   2 +
> >  2 files changed, 367 insertions(+)
> >  create mode 100644 
> > IB-ipoib-move-back-the-IB-LL-address-into-the-hard-header.patch
> > 
> I wonder if this is the problem I have been facing with NFS over IB
> since upgrading to 4.x. Stability and performance problems so bad that
> I have temperately disable this storage and configured NFS over the
> ethernet nics instead.

The 4.4.21-69 kernel package in pvetest should include the fix if you
want to try it (The last download link in the #927 bugzilla entry should
still be valid, too, and is a rebuild of the -68 rel)


signature.asc
Description: Digital signature
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH kernel] Fix #927: add IPoIB performance regression fix

[Michael Rasmussen]

On Thu, 13 Oct 2016 16:49:30 +0200
Wolfgang Bumiller  wrote:

> Fixes kernel bug #111921
> ---
>  ...ck-the-IB-LL-address-into-the-hard-header.patch | 365 
> +
>  Makefile   |   2 +
>  2 files changed, 367 insertions(+)
>  create mode 100644 
> IB-ipoib-move-back-the-IB-LL-address-into-the-hard-header.patch
> 
I wonder if this is the problem I have been facing with NFS over IB
since upgrading to 4.x. Stability and performance problems so bad that
I have temperately disable this storage and configured NFS over the
ethernet nics instead.

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael  rasmussen  cc
http://pgp.mit.edu:11371/pks/lookup?op=get=0xD3C9A00E
mir  datanom  net
http://pgp.mit.edu:11371/pks/lookup?op=get=0xE501F51C
mir  miras  org
http://pgp.mit.edu:11371/pks/lookup?op=get=0xE3E80917
--
/usr/games/fortune -es says:
It's gonna be alright,
It's almost midnight,
And I've got two more bottles of wine.


pgpoP8SRyqNEd.pgp
Description: OpenPGP digital signature
___
pve-devel mailing list
pve-devel@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH kernel] Fix #927: add IPoIB performance regression fix

[Wolfgang Bumiller]

Fixes kernel bug #111921
---
 ...ck-the-IB-LL-address-into-the-hard-header.patch | 365 +
 Makefile   |   2 +
 2 files changed, 367 insertions(+)
 create mode 100644 
IB-ipoib-move-back-the-IB-LL-address-into-the-hard-header.patch

diff --git a/IB-ipoib-move-back-the-IB-LL-address-into-the-hard-header.patch 
b/IB-ipoib-move-back-the-IB-LL-address-into-the-hard-header.patch
new file mode 100644
index 000..5b58edd
--- /dev/null
+++ b/IB-ipoib-move-back-the-IB-LL-address-into-the-hard-header.patch
@@ -0,0 +1,365 @@
+From patchwork Wed Oct 12 14:30:30 2016
+Content-Type: text/plain; charset="utf-8"
+MIME-Version: 1.0
+Content-Transfer-Encoding: 7bit
+Subject: [v2] IB/ipoib: move back IB LL address into the hard header
+From: Paolo Abeni 
+X-Patchwork-Id: 681344
+X-Patchwork-Delegate: da...@davemloft.net
+Message-Id: 
<60efcf739ce3d45a01a7127dbaf7fe366e5ddce4.1476264804.git.pab...@redhat.com>
+To: linux-r...@vger.kernel.org
+Cc: Doug Ledford , Sean Hefty ,
+ Hal Rosenstock ,
+ Jason Gunthorpe , net...@vger.kernel.org
+Date: Wed, 12 Oct 2016 16:30:30 +0200
+
+After the commit 9207f9d45b0a ("net: preserve IP control block
+during GSO segmentation"), the GSO CB and the IPoIB CB conflict.
+That destroy the IPoIB address information cached there,
+causing a severe performance regression, as better described here:
+
+http://marc.info/?l=linux-kernel=146787279825501=2
+
+This change moves the data cached by the IPoIB driver from the
+skb control lock into the IPoIB hard header, as done before
+the commit 936d7de3d736 ("IPoIB: Stop lying about hard_header_len
+and use skb->cb to stash LL addresses").
+In order to avoid GRO issue, on packet reception, the IPoIB driver
+stash into the skb a dummy pseudo header, so that the received
+packets have actually a hard header matching the declared length.
+To avoid changing the connected mode maximum mtu, the allocated 
+head buffer size is increased by the pseudo header length.
+
+After this commit, IPoIB performances are back to pre-regression
+value.
+
+v1 -> v2: avoid changing the max mtu, increasing the head buf size
+
+Fixes: 9207f9d45b0a ("net: preserve IP control block during GSO segmentation")
+Signed-off-by: Paolo Abeni 
+---
+ drivers/infiniband/ulp/ipoib/ipoib.h   | 20 +++---
+ drivers/infiniband/ulp/ipoib/ipoib_cm.c| 15 +++
+ drivers/infiniband/ulp/ipoib/ipoib_ib.c| 12 +++---
+ drivers/infiniband/ulp/ipoib/ipoib_main.c  | 54 --
+ drivers/infiniband/ulp/ipoib/ipoib_multicast.c |  6 ++-
+ 5 files changed, 64 insertions(+), 43 deletions(-)
+
+diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h 
b/drivers/infiniband/ulp/ipoib/ipoib.h
+index 9dbfcc0..5ff64af 100644
+--- a/drivers/infiniband/ulp/ipoib/ipoib.h
 b/drivers/infiniband/ulp/ipoib/ipoib.h
+@@ -63,6 +63,8 @@ enum ipoib_flush_level {
+ 
+ enum {
+   IPOIB_ENCAP_LEN   = 4,
++  IPOIB_PSEUDO_LEN  = 20,
++  IPOIB_HARD_LEN= IPOIB_ENCAP_LEN + IPOIB_PSEUDO_LEN,
+ 
+   IPOIB_UD_HEAD_SIZE= IB_GRH_BYTES + IPOIB_ENCAP_LEN,
+   IPOIB_UD_RX_SG= 2, /* max buffer needed for 4K mtu */
+@@ -134,15 +136,21 @@ struct ipoib_header {
+   u16 reserved;
+ };
+ 
+-struct ipoib_cb {
+-  struct qdisc_skb_cb qdisc_cb;
+-  u8  hwaddr[INFINIBAND_ALEN];
++struct ipoib_pseudo_header {
++  u8  hwaddr[INFINIBAND_ALEN];
+ };
+ 
+-static inline struct ipoib_cb *ipoib_skb_cb(const struct sk_buff *skb)
++static inline void skb_add_pseudo_hdr(struct sk_buff *skb)
+ {
+-  BUILD_BUG_ON(sizeof(skb->cb) < sizeof(struct ipoib_cb));
+-  return (struct ipoib_cb *)skb->cb;
++  char *data = skb_push(skb, IPOIB_PSEUDO_LEN);
++
++  /*
++   * only the ipoib header is present now, make room for a dummy
++   * pseudo header and set skb field accordingly
++   */
++  memset(data, 0, IPOIB_PSEUDO_LEN);
++  skb_reset_mac_header(skb);
++  skb_pull(skb, IPOIB_HARD_LEN);
+ }
+ 
+ /* Used for all multicast joins (broadcast, IPv4 mcast and IPv6 mcast) */
+diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c 
b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
+index 4ad297d..339a1ee 100644
+--- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
 b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
+@@ -63,6 +63,8 @@ MODULE_PARM_DESC(cm_data_debug_level,
+ #define IPOIB_CM_RX_DELAY   (3 * 256 * HZ)
+ #define IPOIB_CM_RX_UPDATE_MASK (0x3)
+ 
++#define IPOIB_CM_RX_RESERVE (ALIGN(IPOIB_HARD_LEN, 16) - IPOIB_ENCAP_LEN)
++
+ static struct ib_qp_attr ipoib_cm_err_attr = {
+   .qp_state = IB_QPS_ERR
+ };
+@@ -146,15 +148,15 @@ static struct sk_buff *ipoib_cm_alloc_rx_skb(struct 
net_device *dev,
+   struct sk_buff *skb;
+   int i;
+ 
+-  skb =