[Kernel-packages] [Bug 1853197] Re: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
Tested 5.3.0-25-generic on Eoan and it fixes the memory leak there as well. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1853197 Title: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection Status in linux package in Ubuntu: Invalid Status in linux source package in Bionic: Fix Committed Status in linux source package in Disco: Fix Committed Status in linux source package in Eoan: Fix Committed Bug description: [SRU Justification] == Impact == An upstream change in v4.11 made xfrm loose memory (8 pages per ipsec connection). This was fixed in v5.4 by: commit 86c6739eda7d "xfrm: Fix memleak on xfrm state destroy" == Fix == Pick the upstream fix into all affected series. == Testcase == see below == Risk of Regression == Low, the change adds a single memory release case in one driver. The effect can be verified. --- Ubuntu linux distro, 4.15.0-62 kernel, server platform. This OS is used as an IPSec VPN gateway. It serves up to several hundred concurrent connections In an attempt to upgrade from the 4.4 kernel to 4.15, the team noticed that VPN gateway VMs were running out of physical memory after 12-48 hours, depending on load. Attachments from a server machine in this state in attached leakinfo.txt output of free -t output of /proc/meminfo in out of memory condition output of /slabtop -o -sc /sys/kernel/debug/page_owner sorted and aggregated after server ran for 12 hrs and ran out of memory Patches for 4.15 and 5.4 Highlight from page_owner, we can see the leak is a buffer associated with the ipsec impelementation. Each connection leaks 32k of memory via alloc_page with order=3 100960 times: Page allocated via order 3, mask 0x1085220(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP) get_page_from_freelist+0xd64/0x1250 __alloc_pages_nodemask+0x11c/0x2e0 alloc_pages_current+0x6a/0xe0 skb_page_frag_refill+0x71/0x100 esp_output_head+0x265/0x3e0 [esp4] esp_output+0xbc/0x180 [esp4] xfrm_output_resume+0x179/0x530 xfrm_output+0x8e/0x230 xfrm4_output_finish+0x2b/0x30 __xfrm4_output+0x3a/0x50 xfrm4_output+0x43/0xc0 ip_forward_finish+0x51/0x80 ip_forward+0x38a/0x480 ip_rcv_finish+0x122/0x410 ip_rcv+0x292/0x360 __netif_receive_skb_core+0x815/0xbd0 Patch to fix this issue in 4.15 (tested and verified on same server exhibiting above leak): diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index 728272f..7842f83 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -451,6 +451,10 @@ static void xfrm_state_gc_destroy(struct xfrm_state *x) } xfrm_dev_state_free(x); security_xfrm_state_free(x); + + if(x->xfrag.page) + put_page(x->xfrag.page); + kfree(x); } Patch for master branch (5.4 I believe) from Paul Wouters (p...@nohats.ca) diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index c6f3c4a1bd99..f3423562d933 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -495,6 +495,8 @@ static void ___xfrm_state_destroy(struct xfrm_state *x) x->type->destructor(x); xfrm_put_type(x->type); } + if (x->xfrag.page) + put_page(x->xfrag.page); xfrm_dev_state_free(x); security_xfrm_state_free(x); xfrm_state_free(x); Severity: Critical - we are unable to use any kernel later than 4.11, and are sticking with 4.4 in production. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1853197] Re: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
Does it help when we test disco and eoan as well? The test case is very easy and those kernels are affected as well. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1853197 Title: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection Status in linux package in Ubuntu: Invalid Status in linux source package in Bionic: Fix Committed Status in linux source package in Disco: Fix Committed Status in linux source package in Eoan: Fix Committed Bug description: [SRU Justification] == Impact == An upstream change in v4.11 made xfrm loose memory (8 pages per ipsec connection). This was fixed in v5.4 by: commit 86c6739eda7d "xfrm: Fix memleak on xfrm state destroy" == Fix == Pick the upstream fix into all affected series. == Testcase == see below == Risk of Regression == Low, the change adds a single memory release case in one driver. The effect can be verified. --- Ubuntu linux distro, 4.15.0-62 kernel, server platform. This OS is used as an IPSec VPN gateway. It serves up to several hundred concurrent connections In an attempt to upgrade from the 4.4 kernel to 4.15, the team noticed that VPN gateway VMs were running out of physical memory after 12-48 hours, depending on load. Attachments from a server machine in this state in attached leakinfo.txt output of free -t output of /proc/meminfo in out of memory condition output of /slabtop -o -sc /sys/kernel/debug/page_owner sorted and aggregated after server ran for 12 hrs and ran out of memory Patches for 4.15 and 5.4 Highlight from page_owner, we can see the leak is a buffer associated with the ipsec impelementation. Each connection leaks 32k of memory via alloc_page with order=3 100960 times: Page allocated via order 3, mask 0x1085220(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP) get_page_from_freelist+0xd64/0x1250 __alloc_pages_nodemask+0x11c/0x2e0 alloc_pages_current+0x6a/0xe0 skb_page_frag_refill+0x71/0x100 esp_output_head+0x265/0x3e0 [esp4] esp_output+0xbc/0x180 [esp4] xfrm_output_resume+0x179/0x530 xfrm_output+0x8e/0x230 xfrm4_output_finish+0x2b/0x30 __xfrm4_output+0x3a/0x50 xfrm4_output+0x43/0xc0 ip_forward_finish+0x51/0x80 ip_forward+0x38a/0x480 ip_rcv_finish+0x122/0x410 ip_rcv+0x292/0x360 __netif_receive_skb_core+0x815/0xbd0 Patch to fix this issue in 4.15 (tested and verified on same server exhibiting above leak): diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index 728272f..7842f83 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -451,6 +451,10 @@ static void xfrm_state_gc_destroy(struct xfrm_state *x) } xfrm_dev_state_free(x); security_xfrm_state_free(x); + + if(x->xfrag.page) + put_page(x->xfrag.page); + kfree(x); } Patch for master branch (5.4 I believe) from Paul Wouters (p...@nohats.ca) diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index c6f3c4a1bd99..f3423562d933 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -495,6 +495,8 @@ static void ___xfrm_state_destroy(struct xfrm_state *x) x->type->destructor(x); xfrm_put_type(x->type); } + if (x->xfrag.page) + put_page(x->xfrag.page); xfrm_dev_state_free(x); security_xfrm_state_free(x); xfrm_state_free(x); Severity: Critical - we are unable to use any kernel later than 4.11, and are sticking with 4.4 in production. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1853197] Re: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
it is running for five days and memory consumption looks normal (not leaking) ** Changed in: linux (Ubuntu Bionic) Status: Fix Committed => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1853197 Title: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection Status in linux package in Ubuntu: Invalid Status in linux source package in Bionic: Confirmed Status in linux source package in Disco: Fix Committed Status in linux source package in Eoan: Fix Committed Bug description: [SRU Justification] == Impact == An upstream change in v4.11 made xfrm loose memory (8 pages per ipsec connection). This was fixed in v5.4 by: commit 86c6739eda7d "xfrm: Fix memleak on xfrm state destroy" == Fix == Pick the upstream fix into all affected series. == Testcase == see below == Risk of Regression == Low, the change adds a single memory release case in one driver. The effect can be verified. --- Ubuntu linux distro, 4.15.0-62 kernel, server platform. This OS is used as an IPSec VPN gateway. It serves up to several hundred concurrent connections In an attempt to upgrade from the 4.4 kernel to 4.15, the team noticed that VPN gateway VMs were running out of physical memory after 12-48 hours, depending on load. Attachments from a server machine in this state in attached leakinfo.txt output of free -t output of /proc/meminfo in out of memory condition output of /slabtop -o -sc /sys/kernel/debug/page_owner sorted and aggregated after server ran for 12 hrs and ran out of memory Patches for 4.15 and 5.4 Highlight from page_owner, we can see the leak is a buffer associated with the ipsec impelementation. Each connection leaks 32k of memory via alloc_page with order=3 100960 times: Page allocated via order 3, mask 0x1085220(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP) get_page_from_freelist+0xd64/0x1250 __alloc_pages_nodemask+0x11c/0x2e0 alloc_pages_current+0x6a/0xe0 skb_page_frag_refill+0x71/0x100 esp_output_head+0x265/0x3e0 [esp4] esp_output+0xbc/0x180 [esp4] xfrm_output_resume+0x179/0x530 xfrm_output+0x8e/0x230 xfrm4_output_finish+0x2b/0x30 __xfrm4_output+0x3a/0x50 xfrm4_output+0x43/0xc0 ip_forward_finish+0x51/0x80 ip_forward+0x38a/0x480 ip_rcv_finish+0x122/0x410 ip_rcv+0x292/0x360 __netif_receive_skb_core+0x815/0xbd0 Patch to fix this issue in 4.15 (tested and verified on same server exhibiting above leak): diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index 728272f..7842f83 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -451,6 +451,10 @@ static void xfrm_state_gc_destroy(struct xfrm_state *x) } xfrm_dev_state_free(x); security_xfrm_state_free(x); + + if(x->xfrag.page) + put_page(x->xfrag.page); + kfree(x); } Patch for master branch (5.4 I believe) from Paul Wouters (p...@nohats.ca) diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index c6f3c4a1bd99..f3423562d933 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -495,6 +495,8 @@ static void ___xfrm_state_destroy(struct xfrm_state *x) x->type->destructor(x); xfrm_put_type(x->type); } + if (x->xfrag.page) + put_page(x->xfrag.page); xfrm_dev_state_free(x); security_xfrm_state_free(x); xfrm_state_free(x); Severity: Critical - we are unable to use any kernel later than 4.11, and are sticking with 4.4 in production. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1854315] Re: IPSec / xfrm memory leak found
some additional hints and reports. https://lore.kernel.org/netdev/CAMnf+Ph-Bx=wzxtpxuc8h6vxtdf-7z52ywdrt9gm6wn0x14...@mail.gmail.com/ https://lore.kernel.org/netdev/CAMnf+PjGq2qsZzg=+H5Z5kO+PSQbo=R0MHW5rv1CWrqoS=b...@mail.gmail.com/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1854315 Title: IPSec / xfrm memory leak found Status in linux package in Ubuntu: Confirmed Bug description: Hi everybody. there is a memory leak in the current kernels since 4.15 (maybe olders as well) in combination with IPSec. I verified it with kernel linux- meta 4.15.0.70.72 on Ubuntu 18.04. As Strongswan is used and users login / logout, being connected and doing traffic memory gets lost. After a while no memory is left over and OOM killer starts it work. After some debugging and asking in the Strongswan irc channel I was informed that there was a memory leak found in net/xfrm/xfrm_state.c which might be related. Here is the commit to this fix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=86c6739eda7d2a03f2db30cbee67a5fb81afa8ba I applied this patch against the linux-meta 4.15.0.70.72 and it fixes the issue. Tested and verified on and with: Ubuntu 18.04.3 LTS strongswan-5.6.2-1ubuntu2.4 linux-image-generic-4.15.0.70.72 Here are some additional information: https://lore.kernel.org/netdev/2019062832.gp13...@gauss3.secunet.de/ https://marc.info/?l=linux-netdev=157405892918311=2 it also is reproducible with hwe and hwe-edge kernel as well as with other strongswan version. I also backported latest Ubuntu version of strongswan to 18.04 which has same behavior. On a completely different system (Gentoo) it also is the case. So I'm really sure that this is related to the reported kernel thing which is already solve in current mainline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1854315/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1854315] [NEW] IPSec / xfrm memory leak found
Public bug reported: Hi everybody. there is a memory leak in the current kernels since 4.15 (maybe olders as well) in combination with IPSec. I verified it with kernel linux-meta 4.15.0.70.72 on Ubuntu 18.04. As Strongswan is used and users login / logout, being connected and doing traffic memory gets lost. After a while no memory is left over and OOM killer starts it work. After some debugging and asking in the Strongswan irc channel I was informed that there was a memory leak found in net/xfrm/xfrm_state.c which might be related. Here is the commit to this fix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=86c6739eda7d2a03f2db30cbee67a5fb81afa8ba I applied this patch against the linux-meta 4.15.0.70.72 and it fixes the issue. Tested and verified on and with: Ubuntu 18.04.3 LTS strongswan-5.6.2-1ubuntu2.4 linux-image-generic-4.15.0.70.72 Here are some additional information: https://lore.kernel.org/netdev/2019062832.gp13...@gauss3.secunet.de/ https://marc.info/?l=linux-netdev=157405892918311=2 it also is reproducible with hwe and hwe-edge kernel as well as with other strongswan version. I also backported latest Ubuntu version of strongswan to 18.04 which has same behavior. On a completely different system (Gentoo) it also is the case. So I'm really sure that this is related to the reported kernel thing which is already solve in current mainline ** Affects: linux-meta (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1854315 Title: IPSec / xfrm memory leak found Status in linux-meta package in Ubuntu: New Bug description: Hi everybody. there is a memory leak in the current kernels since 4.15 (maybe olders as well) in combination with IPSec. I verified it with kernel linux- meta 4.15.0.70.72 on Ubuntu 18.04. As Strongswan is used and users login / logout, being connected and doing traffic memory gets lost. After a while no memory is left over and OOM killer starts it work. After some debugging and asking in the Strongswan irc channel I was informed that there was a memory leak found in net/xfrm/xfrm_state.c which might be related. Here is the commit to this fix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=86c6739eda7d2a03f2db30cbee67a5fb81afa8ba I applied this patch against the linux-meta 4.15.0.70.72 and it fixes the issue. Tested and verified on and with: Ubuntu 18.04.3 LTS strongswan-5.6.2-1ubuntu2.4 linux-image-generic-4.15.0.70.72 Here are some additional information: https://lore.kernel.org/netdev/2019062832.gp13...@gauss3.secunet.de/ https://marc.info/?l=linux-netdev=157405892918311=2 it also is reproducible with hwe and hwe-edge kernel as well as with other strongswan version. I also backported latest Ubuntu version of strongswan to 18.04 which has same behavior. On a completely different system (Gentoo) it also is the case. So I'm really sure that this is related to the reported kernel thing which is already solve in current mainline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/1854315/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1854315] Re: IPSec / xfrm memory leak found
package was not taken correctly ** Package changed: linux (Ubuntu) => linux-meta (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-meta in Ubuntu. https://bugs.launchpad.net/bugs/1854315 Title: IPSec / xfrm memory leak found Status in linux-meta package in Ubuntu: New Bug description: Hi everybody. there is a memory leak in the current kernels since 4.15 (maybe olders as well) in combination with IPSec. I verified it with kernel linux- meta 4.15.0.70.72 on Ubuntu 18.04. As Strongswan is used and users login / logout, being connected and doing traffic memory gets lost. After a while no memory is left over and OOM killer starts it work. After some debugging and asking in the Strongswan irc channel I was informed that there was a memory leak found in net/xfrm/xfrm_state.c which might be related. Here is the commit to this fix: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=86c6739eda7d2a03f2db30cbee67a5fb81afa8ba I applied this patch against the linux-meta 4.15.0.70.72 and it fixes the issue. Tested and verified on and with: Ubuntu 18.04.3 LTS strongswan-5.6.2-1ubuntu2.4 linux-image-generic-4.15.0.70.72 Here are some additional information: https://lore.kernel.org/netdev/2019062832.gp13...@gauss3.secunet.de/ https://marc.info/?l=linux-netdev=157405892918311=2 it also is reproducible with hwe and hwe-edge kernel as well as with other strongswan version. I also backported latest Ubuntu version of strongswan to 18.04 which has same behavior. On a completely different system (Gentoo) it also is the case. So I'm really sure that this is related to the reported kernel thing which is already solve in current mainline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/1854315/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
