lhotari commented on pull request #11414: URL: https://github.com/apache/pulsar/pull/11414#issuecomment-884501342
I was now able to capture the error message when OOM killer kicks in. https://gist.github.com/lhotari/b229863e6dadf2a7239a28d9c8ff3a15 is the `dmesg` output. ``` [ 2944.319334] chronyd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [ 2944.319338] CPU: 0 PID: 666 Comm: chronyd Not tainted 5.8.0-1036-azure #38~20.04.1-Ubuntu [ 2944.319339] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018 [ 2944.319340] Call Trace: [ 2944.319350] dump_stack+0x5e/0x74 [ 2944.319353] dump_header+0x4f/0x1eb [ 2944.319355] oom_kill_process.cold+0xb/0x10 [ 2944.319359] out_of_memory.part.0+0x1df/0x430 [ 2944.319361] out_of_memory+0x6d/0xd0 [ 2944.319364] __alloc_pages_slowpath.constprop.0+0xd02/0xe20 [ 2944.319366] ? get_page_from_freelist+0x20c/0x370 [ 2944.319368] __alloc_pages_nodemask+0x2bc/0x2f0 [ 2944.319372] alloc_pages_current+0x87/0xe0 [ 2944.319373] __page_cache_alloc+0x72/0x90 [ 2944.319375] pagecache_get_page+0xbf/0x300 [ 2944.319377] filemap_fault+0x6d2/0xa50 [ 2944.319380] ? page_add_file_rmap+0x14e/0x1c0 [ 2944.319382] ? filemap_map_pages+0x1cf/0x3a0 [ 2944.319386] ext4_filemap_fault+0x32/0x50 [ 2944.319389] __do_fault+0x3c/0x130 [ 2944.319391] do_fault+0x248/0x5f0 [ 2944.319392] __handle_mm_fault+0x5a2/0xbd0 [ 2944.319394] handle_mm_fault+0xb4/0x1e0 [ 2944.319398] do_user_addr_fault+0x1dd/0x430 [ 2944.319401] ? __prepare_exit_to_usermode+0xac/0x200 [ 2944.319404] exc_page_fault+0x80/0x1a0 [ 2944.319406] ? asm_exc_page_fault+0x8/0x30 [ 2944.319408] asm_exc_page_fault+0x1e/0x30 [ 2944.319411] RIP: 0033:0x55d3ff7766e0 [ 2944.319417] Code: Unable to access opcode bytes at RIP 0x55d3ff7766b6. [ 2944.319418] RSP: 002b:00007ffedbe95368 EFLAGS: 00010202 [ 2944.319419] RAX: 000055d3ff7766e0 RBX: 000055d400c1ba90 RCX: 000055d400c1d300 [ 2944.319420] RDX: 000055d3ff789940 RSI: 000055d3ff789940 RDI: 000055d400c1ba90 [ 2944.319421] RBP: 00000000fffffffe R08: 000000000013d518 R09: 0000000060f88517 [ 2944.319422] R10: 0000000000000002 R11: 00007ffedbf6e000 R12: 000055d3ff7898e0 [ 2944.319422] R13: 000055d3ff7754f0 R14: 00007ffedbe95470 R15: 000055d400c1ba90 [ 2944.319425] Mem-Info: [ 2944.319429] active_anon:1683588 inactive_anon:269 isolated_anon:0 active_file:83 inactive_file:0 isolated_file:0 unevictable:6680 dirty:2 writeback:0 slab_reclaimable:15199 slab_unreclaimable:21963 mapped:2270 shmem:618 pagetables:5637 bounce:0 free:26467 free_pcp:0 free_cma:0 [ 2944.319431] Node 0 active_anon:6734352kB inactive_anon:1076kB active_file:332kB inactive_file:0kB unevictable:26720kB isolated(anon):0kB isolated(file):0kB mapped:9080kB dirty:8kB writeback:0kB shmem:2472kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 65536kB writeback_tmp:0kB all_unreclaimable? yes [ 2944.319432] Node 0 DMA free:14884kB min:148kB low:184kB high:220kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 2944.319435] lowmem_reserve[]: 0 943 6930 6930 6930 [ 2944.319437] Node 0 DMA32 free:32988kB min:9172kB low:11464kB high:13756kB reserved_highatomic:0KB active_anon:905992kB inactive_anon:8kB active_file:20kB inactive_file:0kB unevictable:0kB writepending:4kB present:1032128kB managed:966592kB mlocked:0kB kernel_stack:6316kB pagetables:3280kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 2944.319439] lowmem_reserve[]: 0 0 5986 5986 5986 [ 2944.319441] Node 0 Normal free:57996kB min:58256kB low:72820kB high:87384kB reserved_highatomic:0KB active_anon:5828012kB inactive_anon:1068kB active_file:220kB inactive_file:12kB unevictable:26720kB writepending:4kB present:6291456kB managed:6138300kB mlocked:26720kB kernel_stack:16164kB pagetables:19268kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 2944.319443] lowmem_reserve[]: 0 0 0 0 0 [ 2944.319445] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14884kB [ 2944.319450] Node 0 DMA32: 354*4kB (UME) 263*8kB (UME) 205*16kB (UME) 181*32kB (UME) 130*64kB (UME) 57*128kB (UME) 16*256kB (UME) 3*512kB (UME) 0*1024kB 0*2048kB 0*4096kB = 33840kB [ 2944.319455] Node 0 Normal: 3325*4kB (UME) 1767*8kB (UME) 754*16kB (UME) 424*32kB (UME) 89*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 58764kB [ 2944.319461] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 2944.319462] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 2944.319462] 2762 total pagecache pages [ 2944.319464] 0 pages in swap cache [ 2944.319465] Swap cache stats: add 0, delete 0, find 0/0 [ 2944.319465] Free swap = 0kB [ 2944.319466] Total swap = 0kB [ 2944.319467] 1834894 pages RAM [ 2944.319467] 0 pages HighMem/MovableOnly [ 2944.319468] 54694 pages reserved [ 2944.319468] 0 pages hwpoisoned [ 2944.319469] Tasks state (memory values in pages): [ 2944.319469] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 2944.319473] [ 189] 0 189 17498 1957 114688 0 -250 systemd-journal [ 2944.319475] [ 223] 0 223 4711 859 57344 0 -1000 systemd-udevd [ 2944.319477] [ 298] 0 298 1033 676 49152 0 0 hv_kvp_daemon [ 2944.319478] [ 320] 0 320 269 1 28672 0 0 none [ 2944.319481] [ 396] 0 396 72100 6492 118784 0 -1000 multipathd [ 2944.319483] [ 451] 0 451 2076 1512 53248 0 0 haveged [ 2944.319484] [ 531] 100 531 8734 972 77824 0 0 systemd-network [ 2944.319486] [ 533] 101 533 5975 1541 81920 0 0 systemd-resolve [ 2944.319487] [ 647] 0 647 66405 718 102400 0 0 accounts-daemon [ 2944.319489] [ 655] 0 655 2136 582 53248 0 0 cron [ 2944.319491] [ 657] 103 657 1920 890 53248 0 -900 dbus-daemon [ 2944.319492] [ 665] 0 665 22507 423 61440 0 0 irqbalance [ 2944.319494] [ 666] 112 666 1207 304 53248 0 0 chronyd [ 2944.319495] [ 673] 112 673 1174 48 53248 0 0 chronyd [ 2944.319497] [ 674] 0 674 7422 2799 90112 0 0 networkd-dispat [ 2944.319498] [ 679] 104 679 58175 760 86016 0 0 rsyslogd [ 2944.319500] [ 680] 0 680 926107 13169 512000 0 0 provisioner [ 2944.319501] [ 683] 0 683 237560 6075 258048 0 -900 snapd [ 2944.319503] [ 698] 0 698 4153 422 73728 0 0 systemd-logind [ 2944.319504] [ 701] 0 701 950 533 49152 0 0 atd [ 2944.319506] [ 704] 0 704 393768 6977 307200 0 -999 containerd [ 2944.319507] [ 730] 0 730 1840 404 49152 0 0 agetty [ 2944.319509] [ 733] 0 733 1459 413 49152 0 0 agetty [ 2944.319510] [ 735] 0 735 3046 806 61440 0 -1000 sshd [ 2944.319512] [ 795] 0 795 65252 1018 94208 0 0 polkitd [ 2944.319514] [ 868] 0 868 380845 16910 577536 0 -500 dockerd [ 2944.319515] [ 1951] 0 1951 341178 2900 188416 0 0 provjobd [ 2944.319517] [ 1960] 1001 1960 830997 11629 528384 0 0 Runner.Listener [ 2944.319518] [ 1979] 1001 1979 842838 15576 569344 0 0 Runner.Worker [ 2944.319520] [ 6469] 0 6469 29756 302 61440 0 -500 unpigz [ 2944.319522] [ 10928] 0 10928 29756 301 65536 0 -500 unpigz [ 2944.319524] [ 15766] 0 15766 13372 238 53248 0 -500 unpigz [ 2944.319525] [ 18614] 1001 18614 2176 541 49152 0 500 bash [ 2944.319527] [ 18615] 1001 18615 2209 579 57344 0 500 bash [ 2944.319528] [ 18622] 1001 18622 2209 311 53248 0 500 bash [ 2944.319529] [ 18623] 1001 18623 622266 145030 1601536 0 500 java [ 2944.319531] [ 18680] 1001 18680 655 122 45056 0 500 sh [ 2944.319532] [ 18682] 1001 18682 712979 87602 1056768 0 500 java [ 2944.319534] [ 18801] 0 18801 169833 267 135168 0 -500 docker-proxy [ 2944.319535] [ 18809] 0 18809 129223 306 118784 0 -500 docker-proxy [ 2944.319536] [ 18822] 0 18822 177977 632 106496 0 -998 containerd-shim [ 2944.319538] [ 18841] 0 18841 27183 543 77824 0 0 app [ 2944.319539] [ 18955] 0 18955 128935 104 122880 0 -500 docker-proxy [ 2944.319540] [ 18960] 0 18960 128871 184 110592 0 -500 docker-proxy [ 2944.319542] [ 18974] 0 18974 178329 686 114688 0 -998 containerd-shim [ 2944.319544] [ 18994] 0 18994 6729 3325 90112 0 0 supervisord [ 2944.319545] [ 19119] 0 19119 444283 34054 630784 0 0 java [ 2944.319546] [ 19323] 0 19323 149704 354 131072 0 -500 docker-proxy [ 2944.319548] [ 19330] 0 19330 128871 103 122880 0 -500 docker-proxy [ 2944.319549] [ 19345] 0 19345 178393 637 114688 0 -998 containerd-shim [ 2944.319551] [ 19366] 0 19366 6729 3323 94208 0 0 supervisord [ 2944.319552] [ 19456] 0 19456 444029 76288 1028096 0 0 java [ 2944.319553] [ 19798] 0 19798 128871 203 118784 0 -500 docker-proxy [ 2944.319554] [ 19803] 0 19803 149352 205 122880 0 -500 docker-proxy [ 2944.319556] [ 19819] 0 19819 178329 614 114688 0 -998 containerd-shim [ 2944.319557] [ 19840] 0 19840 6728 3327 90112 0 0 supervisord [ 2944.319558] [ 19957] 0 19957 669309 296189 2813952 0 0 java [ 2944.319560] [ 20079] 0 20079 149352 154 122880 0 -500 docker-proxy [ 2944.319561] [ 20085] 0 20085 149352 207 122880 0 -500 docker-proxy [ 2944.319562] [ 20101] 0 20101 178329 636 114688 0 -998 containerd-shim [ 2944.319564] [ 20121] 0 20121 6727 3323 94208 0 0 supervisord [ 2944.319565] [ 20348] 0 20348 668496 279593 2732032 0 0 java [ 2944.319566] [ 20581] 0 20581 128935 190 114688 0 -500 docker-proxy [ 2944.319568] [ 20587] 0 20587 108390 210 106496 0 -500 docker-proxy [ 2944.319569] [ 20606] 0 20606 149352 139 122880 0 -500 docker-proxy [ 2944.319570] [ 20612] 0 20612 129223 393 118784 0 -500 docker-proxy [ 2944.319571] [ 20633] 0 20633 178393 739 118784 0 -998 containerd-shim [ 2944.319573] [ 20654] 0 20654 6728 3338 98304 0 0 supervisord [ 2944.319574] [ 20774] 0 20774 491367 120145 1232896 0 0 java [ 2944.319576] [ 21260] 0 21260 1021 26 45056 0 0 tail [ 2944.319577] [ 21287] 0 21287 128871 189 118784 0 -500 docker-proxy [ 2944.319579] [ 21292] 0 21292 108390 181 102400 0 -500 docker-proxy [ 2944.319580] [ 21307] 0 21307 149352 107 135168 0 -500 docker-proxy [ 2944.319581] [ 21312] 0 21312 128871 164 114688 0 -500 docker-proxy [ 2944.319583] [ 21328] 0 21328 178329 698 118784 0 -998 containerd-shim [ 2944.319584] [ 21349] 0 21349 6729 3332 86016 0 0 supervisord [ 2944.319585] [ 21459] 0 21459 457759 90099 1150976 0 0 java [ 2944.319587] [ 21836] 0 21836 128935 108 118784 0 -500 docker-proxy [ 2944.319588] [ 21841] 0 21841 149352 107 114688 0 -500 docker-proxy [ 2944.319590] [ 21858] 0 21858 178329 760 110592 0 -998 containerd-shim [ 2944.319591] [ 21878] 0 21878 6728 3326 98304 0 0 supervisord [ 2944.319593] [ 21983] 0 21983 1080407 403337 3780608 0 0 pulsar-presto-d [ 2944.319594] [ 22257] 0 22257 1021 26 49152 0 0 tail [ 2944.319596] [ 22278] 0 22278 1021 27 45056 0 0 tail [ 2944.319597] [ 22296] 0 22296 1021 25 53248 0 0 tail [ 2944.319598] [ 24733] 0 24733 2013 482 45056 0 0 jobkeepalive [ 2944.319600] [ 25144] 0 25144 431677 17433 425984 0 0 java [ 2944.319601] [ 25197] 1001 25197 655 107 45056 0 500 sh [ 2944.319603] [ 25198] 1001 25198 2637 636 57344 0 500 ps [ 2944.319604] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/runner-provisioner.service,task=java,pid=18623,uid=1001 [ 2944.319676] Out of memory: Killed process 18623 (java) total-vm:2489064kB, anon-rss:580120kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:1564kB oom_score_adj:500 [ 2944.393513] oom_reaper: reaped process 18623 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
