Public bug reported:
Ubuntu linux distro, 4.15.0-62 kernel, server platform.
This OS is used as an IPSec VPN gateway. It serves up to several hundred
concurrent connections
In an attempt to upgrade from the 4.4 kernel to 4.15, the team noticed
that VPN gateway VMs were running out of physical memory after 12-48
hours, depending on load.
Attachments from a server machine in this state in attached leakinfo.txt
output of free -t
output of /proc/meminfo in out of memory condition
output of /slabtop -o -sc
/sys/kernel/debug/page_owner sorted and aggregated after server ran for 12 hrs
and ran out of memory
Patches for 4.15 and 5.4
Highlight from page_owner, we can see the leak is a buffer associated
with the ipsec impelementation. Each connection leaks 32k of memory via
alloc_page with order=3
Page allocated via order 3, mask
0x1085220(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP)
get_page_from_freelist+0xd64/0x1250
__alloc_pages_nodemask+0x11c/0x2e0
alloc_pages_current+0x6a/0xe0
skb_page_frag_refill+0x71/0x100
esp_output_head+0x265/0x3e0 [esp4]
esp_output+0xbc/0x180 [esp4]
xfrm_output_resume+0x179/0x530
xfrm_output+0x8e/0x230
xfrm4_output_finish+0x2b/0x30
__xfrm4_output+0x3a/0x50
xfrm4_output+0x43/0xc0
ip_forward_finish+0x51/0x80
ip_forward+0x38a/0x480
ip_rcv_finish+0x122/0x410
ip_rcv+0x292/0x360
__netif_receive_skb_core+0x815/0xbd0
Patch to fix this issue in 4.15 (tested and verified on same server exhibiting
above leak):
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index 728272f..7842f83 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -451,6 +451,10 @@ static void xfrm_state_gc_destroy(struct xfrm_state *x)
}
xfrm_dev_state_free(x);
security_xfrm_state_free(x);
+
+ if(x->xfrag.page)
+ put_page(x->xfrag.page);
+
kfree(x);
}
Patch for master branch (5.4 I believe) from Paul Wouters ([email protected])
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index c6f3c4a1bd99..f3423562d933 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -495,6 +495,8 @@ static void ___xfrm_state_destroy(struct xfrm_state *x)
x->type->destructor(x);
xfrm_put_type(x->type);
}
+ if (x->xfrag.page)
+ put_page(x->xfrag.page);
xfrm_dev_state_free(x);
security_xfrm_state_free(x);
xfrm_state_free(x);
Severity: Critical - we are unable to use any kernel later than 4.11,
and are sticking with 4.4 in production.
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: ipsec kernel kernel-bug leak linux memory vpn
** Attachment added: "additional data and patches"
https://bugs.launchpad.net/bugs/1853197/+attachment/5306500/+files/leakinfo.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1853197
Title:
Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
