Update OOM triggered tonight. What i do not understand is that when we sum all pages in use, we get 227935. Dump shows ~1M pages of RAM, and only 228k are used.
[Fri Nov 4 03:57:05 2016] sh invoked oom-killer: gfp_mask=0x2000d0, order=2, oom_score_adj=0 [Fri Nov 4 03:57:05 2016] sh cpuset=/ mems_allowed=0 [Fri Nov 4 03:57:05 2016] CPU: 3 PID: 21020 Comm: sh Not tainted 3.16.0-4-amd64 #1 Debian 3.16.36-1+deb8u1 [Fri Nov 4 03:57:05 2016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [Fri Nov 4 03:57:05 2016] 0000000000000000 ffffffff81512391 ffff8800a38d8350 0000000000000000 [Fri Nov 4 03:57:05 2016] ffffffff8150ff8d 0000000000000000 ffffffff810d6e3f 0000000000000000 [Fri Nov 4 03:57:05 2016] ffffffff81516d0e 0000000000000200 ffffffff810689d3 ffffffff810c43e4 [Fri Nov 4 03:57:05 2016] Call Trace: [Fri Nov 4 03:57:05 2016] [<ffffffff81512391>] ? dump_stack+0x5d/0x78 [Fri Nov 4 03:57:05 2016] [<ffffffff8150ff8d>] ? dump_header+0x76/0x1e8 [Fri Nov 4 03:57:05 2016] [<ffffffff810d6e3f>] ? smp_call_function_single+0x5f/0xa0 [Fri Nov 4 03:57:05 2016] [<ffffffff81516d0e>] ? mutex_lock+0xe/0x2a [Fri Nov 4 03:57:05 2016] [<ffffffff810689d3>] ? put_online_cpus+0x23/0x80 [Fri Nov 4 03:57:05 2016] [<ffffffff810c43e4>] ? rcu_oom_notify+0xc4/0xe0 [Fri Nov 4 03:57:05 2016] [<ffffffff81153d1c>] ? do_try_to_free_pages+0x4ac/0x520 [Fri Nov 4 03:57:05 2016] [<ffffffff811427dd>] ? oom_kill_process+0x21d/0x370 [Fri Nov 4 03:57:05 2016] [<ffffffff8114239d>] ? find_lock_task_mm+0x3d/0x90 [Fri Nov 4 03:57:05 2016] [<ffffffff81142f43>] ? out_of_memory+0x473/0x4b0 [Fri Nov 4 03:57:05 2016] [<ffffffff81148e0f>] ? __alloc_pages_nodemask+0x9ef/0xb50 [Fri Nov 4 03:57:05 2016] [<ffffffff81065c86>] ? copy_process.part.25+0x116/0x1c50 [Fri Nov 4 03:57:05 2016] [<ffffffffa01e476f>] ? ext4_xattr_get+0x6f/0x2a0 [ext4] [Fri Nov 4 03:57:05 2016] [<ffffffff811ba84f>] ? user_path_at_empty+0x5f/0x90 [Fri Nov 4 03:57:05 2016] [<ffffffff811ff5bf>] ? set_cached_acl+0x2f/0x80 [Fri Nov 4 03:57:05 2016] [<ffffffff81067990>] ? do_fork+0xe0/0x3d0 [Fri Nov 4 03:57:05 2016] [<ffffffff811c054f>] ? dput+0x1f/0x170 [Fri Nov 4 03:57:05 2016] [<ffffffff815188b9>] ? stub_clone+0x69/0x90 [Fri Nov 4 03:57:05 2016] [<ffffffff8151854d>] ? system_call_fast_compare_end+0x10/0x15 [Fri Nov 4 03:57:05 2016] Mem-Info: [Fri Nov 4 03:57:05 2016] Node 0 DMA per-cpu: [Fri Nov 4 03:57:05 2016] CPU 0: hi: 0, btch: 1 usd: 0 [Fri Nov 4 03:57:05 2016] CPU 1: hi: 0, btch: 1 usd: 0 [Fri Nov 4 03:57:05 2016] CPU 2: hi: 0, btch: 1 usd: 0 [Fri Nov 4 03:57:05 2016] CPU 3: hi: 0, btch: 1 usd: 0 [Fri Nov 4 03:57:05 2016] Node 0 DMA32 per-cpu: [Fri Nov 4 03:57:05 2016] CPU 0: hi: 186, btch: 31 usd: 0 [Fri Nov 4 03:57:05 2016] CPU 1: hi: 186, btch: 31 usd: 0 [Fri Nov 4 03:57:05 2016] CPU 2: hi: 186, btch: 31 usd: 0 [Fri Nov 4 03:57:05 2016] CPU 3: hi: 186, btch: 31 usd: 0 [Fri Nov 4 03:57:05 2016] Node 0 Normal per-cpu: [Fri Nov 4 03:57:05 2016] CPU 0: hi: 186, btch: 31 usd: 0 [Fri Nov 4 03:57:05 2016] CPU 1: hi: 186, btch: 31 usd: 0 [Fri Nov 4 03:57:05 2016] CPU 2: hi: 186, btch: 31 usd: 0 [Fri Nov 4 03:57:05 2016] CPU 3: hi: 186, btch: 31 usd: 0 [Fri Nov 4 03:57:05 2016] active_anon:99318 inactive_anon:57243 isolated_anon:17 active_file:24 inactive_file:25 isolated_file:0 unevictable:0 dirty:0 writeback:24 unstable:0 free:34157 slab_reclaimable:12263 slab_unreclaimable:55830 mapped:372 shmem:427 pagetables:2392 bounce:0 free_cma:0 [Fri Nov 4 03:57:05 2016] Node 0 DMA free:15892kB min:264kB low:328kB high:396kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [Fri Nov 4 03:57:05 2016] lowmem_reserve[]: 0 2980 3947 3947 [Fri Nov 4 03:57:05 2016] Node 0 DMA32 free:94024kB min:50840kB low:63548kB high:76260kB active_anon:224840kB inactive_anon:56216kB active_file:104kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129212kB managed:3054736kB mlocked:0kB dirty:0kB writeback:0kB mapped:808kB shmem:844kB slab_reclaimable:31820kB slab_unreclaimable:123564kB kernel_stack:1584kB pagetables:4252kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:162 all_unreclaimable? yes [Fri Nov 4 03:57:05 2016] lowmem_reserve[]: 0 0 966 966 [Fri Nov 4 03:57:05 2016] Node 0 Normal free:26712kB min:16476kB low:20592kB high:24712kB active_anon:172432kB inactive_anon:172756kB active_file:0kB inactive_file:96kB unevictable:0kB isolated(anon):68kB isolated(file):0kB present:1048576kB managed:990004kB mlocked:0kB dirty:0kB writeback:96kB mapped:680kB shmem:864kB slab_reclaimable:17232kB slab_unreclaimable:99756kB kernel_stack:1104kB pagetables:5316kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:148 all_unreclaimable? yes [Fri Nov 4 03:57:05 2016] lowmem_reserve[]: 0 0 0 0 [Fri Nov 4 03:57:05 2016] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (EM) = 15892kB [Fri Nov 4 03:57:05 2016] Node 0 DMA32: 17075*4kB (UEM) 2289*8kB (UEM) 334*16kB (UM) 4*32kB (UMR) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 94196kB [Fri Nov 4 03:57:05 2016] Node 0 Normal: 6349*4kB (UE) 125*8kB (UE) 0*16kB 1*32kB (R) 1*64kB (R) 0*128kB 0*256kB 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 27004kB [Fri Nov 4 03:57:05 2016] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Fri Nov 4 03:57:05 2016] 1066 total pagecache pages [Fri Nov 4 03:57:05 2016] 470 pages in swap cache [Fri Nov 4 03:57:05 2016] Swap cache stats: add 5419659, delete 5419189, find 2257447/4348361 [Fri Nov 4 03:57:05 2016] Free swap = 0kB [Fri Nov 4 03:57:05 2016] Total swap = 1355772kB [Fri Nov 4 03:57:05 2016] 1048445 pages RAM [Fri Nov 4 03:57:05 2016] 0 pages HighMem/MovableOnly [Fri Nov 4 03:57:05 2016] 14643 pages reserved [Fri Nov 4 03:57:05 2016] 0 pages hwpoisoned