[Bug 1853197] Re: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
Tested 4.15 bionic with original use case. Memory leak is resolved. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853197 Title: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853197] Re: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
That fix is in the master branch - can it be backported? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853197 Title: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853197] Re: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
** Description changed: Ubuntu linux distro, 4.15.0-62 kernel, server platform. This OS is used as an IPSec VPN gateway. It serves up to several hundred concurrent connections In an attempt to upgrade from the 4.4 kernel to 4.15, the team noticed that VPN gateway VMs were running out of physical memory after 12-48 hours, depending on load. Attachments from a server machine in this state in attached leakinfo.txt output of free -t output of /proc/meminfo in out of memory condition - output of /slabtop -o -sc + output of /slabtop -o -sc /sys/kernel/debug/page_owner sorted and aggregated after server ran for 12 hrs and ran out of memory Patches for 4.15 and 5.4 Highlight from page_owner, we can see the leak is a buffer associated with the ipsec impelementation. Each connection leaks 32k of memory via alloc_page with order=3 + 100960 times: Page allocated via order 3, mask 0x1085220(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP) - get_page_from_freelist+0xd64/0x1250 - __alloc_pages_nodemask+0x11c/0x2e0 - alloc_pages_current+0x6a/0xe0 - skb_page_frag_refill+0x71/0x100 - esp_output_head+0x265/0x3e0 [esp4] - esp_output+0xbc/0x180 [esp4] - xfrm_output_resume+0x179/0x530 - xfrm_output+0x8e/0x230 - xfrm4_output_finish+0x2b/0x30 - __xfrm4_output+0x3a/0x50 - xfrm4_output+0x43/0xc0 - ip_forward_finish+0x51/0x80 - ip_forward+0x38a/0x480 - ip_rcv_finish+0x122/0x410 - ip_rcv+0x292/0x360 - __netif_receive_skb_core+0x815/0xbd0 + get_page_from_freelist+0xd64/0x1250 + __alloc_pages_nodemask+0x11c/0x2e0 + alloc_pages_current+0x6a/0xe0 + skb_page_frag_refill+0x71/0x100 + esp_output_head+0x265/0x3e0 [esp4] + esp_output+0xbc/0x180 [esp4] + xfrm_output_resume+0x179/0x530 + xfrm_output+0x8e/0x230 + xfrm4_output_finish+0x2b/0x30 + __xfrm4_output+0x3a/0x50 + xfrm4_output+0x43/0xc0 + ip_forward_finish+0x51/0x80 + ip_forward+0x38a/0x480 + ip_rcv_finish+0x122/0x410 + ip_rcv+0x292/0x360 + __netif_receive_skb_core+0x815/0xbd0 Patch to fix this issue in 4.15 (tested and verified on same server exhibiting above leak): diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index 728272f..7842f83 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -451,6 +451,10 @@ static void xfrm_state_gc_destroy(struct xfrm_state *x) - } - xfrm_dev_state_free(x); - security_xfrm_state_free(x); + } + xfrm_dev_state_free(x); + security_xfrm_state_free(x); + + if(x->xfrag.page) + put_page(x->xfrag.page); + - kfree(x); + kfree(x); } - - - Patch for master branch (5.4 I believe) from Paul Wouters (p...@nohats.ca) + Patch for master branch (5.4 I believe) from Paul Wouters + (p...@nohats.ca) diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index c6f3c4a1bd99..f3423562d933 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -495,6 +495,8 @@ static void ___xfrm_state_destroy(struct xfrm_state *x) - x->type->destructor(x); - xfrm_put_type(x->type); - } + x->type->destructor(x); + xfrm_put_type(x->type); + } + if (x->xfrag.page) + put_page(x->xfrag.page); - xfrm_dev_state_free(x); - security_xfrm_state_free(x); - xfrm_state_free(x); - + xfrm_dev_state_free(x); + security_xfrm_state_free(x); + xfrm_state_free(x); Severity: Critical - we are unable to use any kernel later than 4.11, and are sticking with 4.4 in production. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853197 Title: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853197] Re: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
All VPN servers have been rolled back to 4.4 Additional log collection is not possible. Setting status to confirmed. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853197 Title: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1853197] [NEW] Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
Public bug reported: Ubuntu linux distro, 4.15.0-62 kernel, server platform. This OS is used as an IPSec VPN gateway. It serves up to several hundred concurrent connections In an attempt to upgrade from the 4.4 kernel to 4.15, the team noticed that VPN gateway VMs were running out of physical memory after 12-48 hours, depending on load. Attachments from a server machine in this state in attached leakinfo.txt output of free -t output of /proc/meminfo in out of memory condition output of /slabtop -o -sc /sys/kernel/debug/page_owner sorted and aggregated after server ran for 12 hrs and ran out of memory Patches for 4.15 and 5.4 Highlight from page_owner, we can see the leak is a buffer associated with the ipsec impelementation. Each connection leaks 32k of memory via alloc_page with order=3 Page allocated via order 3, mask 0x1085220(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP) get_page_from_freelist+0xd64/0x1250 __alloc_pages_nodemask+0x11c/0x2e0 alloc_pages_current+0x6a/0xe0 skb_page_frag_refill+0x71/0x100 esp_output_head+0x265/0x3e0 [esp4] esp_output+0xbc/0x180 [esp4] xfrm_output_resume+0x179/0x530 xfrm_output+0x8e/0x230 xfrm4_output_finish+0x2b/0x30 __xfrm4_output+0x3a/0x50 xfrm4_output+0x43/0xc0 ip_forward_finish+0x51/0x80 ip_forward+0x38a/0x480 ip_rcv_finish+0x122/0x410 ip_rcv+0x292/0x360 __netif_receive_skb_core+0x815/0xbd0 Patch to fix this issue in 4.15 (tested and verified on same server exhibiting above leak): diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index 728272f..7842f83 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -451,6 +451,10 @@ static void xfrm_state_gc_destroy(struct xfrm_state *x) } xfrm_dev_state_free(x); security_xfrm_state_free(x); + + if(x->xfrag.page) + put_page(x->xfrag.page); + kfree(x); } Patch for master branch (5.4 I believe) from Paul Wouters (p...@nohats.ca) diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index c6f3c4a1bd99..f3423562d933 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -495,6 +495,8 @@ static void ___xfrm_state_destroy(struct xfrm_state *x) x->type->destructor(x); xfrm_put_type(x->type); } + if (x->xfrag.page) + put_page(x->xfrag.page); xfrm_dev_state_free(x); security_xfrm_state_free(x); xfrm_state_free(x); Severity: Critical - we are unable to use any kernel later than 4.11, and are sticking with 4.4 in production. ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: ipsec kernel kernel-bug leak linux memory vpn ** Attachment added: "additional data and patches" https://bugs.launchpad.net/bugs/1853197/+attachment/5306500/+files/leakinfo.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853197 Title: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
