When investigating on the affected 32bit ARM system, we have observed that Go runtime would make a request to allocate system memory for it's arena allocator, using a size that was questionable. Even most trivial binary would request 150MB, using mmap(.., PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_PRIVATE..). At the same time, the system would report plenty of available memory, because although the request was satisfied, only few pages were eventually mapped, because the actual use of the mapped region (as reported by pmap -x) was no more than a few MB.
I was able to craft a program that would issue mmap() with large sizes, use only a piece of mapped area and eventually break other programs. This would suggest we would exhaust overcommit limits globally. In the end we managed to reproduce the problem on the affected system using a trivial piece of Go code. The breaking change was enabling -buildmode=pie (like we do for snapd). The program would fail in runtime initialization with `fatal error: runtime: out of memory`. Relevant strace output: brk(NULL) = 0x7f6b0000 mmap2(0x7f700000, 807411712, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f700000 mmap2(0x7f900000, 150536192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) fatal error: runtime: out of memory When a PIE binary was loaded on the system, auxv showed relatively high entry address (0x7f.....). At the same time, brk() would also return high address values (seen in strace output above). Go runtime probes the start address of brk and calculates the size of the arenas. We suspect, that the sizes are miscalculated, resulting in a large mmap() request, thus contributing to the problem. We did a rundown of a different Go versions. We've verified 1.7 and 1.10 to have the problem, 1.11 was seemingly fixed. I've bisected Go versions between 1.10beta2 and 1.11beta1, found this commit https://github.com/golang/go/commit/c0392d2e7fbdcd38aafb959e94daf6bbafe2e4e9 to be the first one when the problem is 'fixed' (or not triggered on that particular kernel + userland combination). The observed mmap() calls request a sane amount of memory, 262kB instead of 150MB like the broken versions did. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1822738 Title: memleak in 2.38+ ? To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1822738/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
