lhotari commented on pull request #11414:
URL: https://github.com/apache/pulsar/pull/11414#issuecomment-884503897
another oom kill
```
[ 5025.370395] systemd-resolve invoked oom-killer:
gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[ 5025.370399] CPU: 0 PID: 533 Comm: systemd-resolve Not tainted
5.8.0-1036-azure #38~20.04.1-Ubuntu
[ 5025.370400] Hardware name: Microsoft Corporation Virtual Machine/Virtual
Machine, BIOS 090008 12/07/2018
[ 5025.370401] Call Trace:
[ 5025.370411] dump_stack+0x5e/0x74
[ 5025.370414] dump_header+0x4f/0x1eb
[ 5025.370416] oom_kill_process.cold+0xb/0x10
[ 5025.370421] out_of_memory.part.0+0x1df/0x430
[ 5025.370422] out_of_memory+0x6d/0xd0
[ 5025.370426] __alloc_pages_slowpath.constprop.0+0xd02/0xe20
[ 5025.370428] ? get_page_from_freelist+0x20c/0x370
[ 5025.370430] __alloc_pages_nodemask+0x2bc/0x2f0
[ 5025.370433] alloc_pages_current+0x87/0xe0
[ 5025.370435] __page_cache_alloc+0x72/0x90
[ 5025.370437] pagecache_get_page+0xbf/0x300
[ 5025.370439] filemap_fault+0x6d2/0xa50
[ 5025.370442] ? page_add_file_rmap+0x14e/0x1c0
[ 5025.370446] ? shmem_getpage_gfp+0xe8/0x8d0
[ 5025.370448] ? xas_load+0x9/0x80
[ 5025.370449] ? xas_find+0x183/0x1b0
[ 5025.370451] ? filemap_map_pages+0x1cf/0x3a0
[ 5025.370455] ext4_filemap_fault+0x32/0x50
[ 5025.370458] __do_fault+0x3c/0x130
[ 5025.370460] do_fault+0x248/0x5f0
[ 5025.370464] ? rcu_nocb_try_bypass+0x225/0x370
[ 5025.370465] __handle_mm_fault+0x5a2/0xbd0
[ 5025.370467] handle_mm_fault+0xb4/0x1e0
[ 5025.370471] do_user_addr_fault+0x1dd/0x430
[ 5025.370474] ? __prepare_exit_to_usermode+0x76/0x200
[ 5025.370477] exc_page_fault+0x80/0x1a0
[ 5025.370479] ? asm_exc_page_fault+0x8/0x30
[ 5025.370480] asm_exc_page_fault+0x1e/0x30
[ 5025.370483] RIP: 0033:0x7fb1d680b08c
[ 5025.370486] Code: 82 29 c8 c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa
89 f9 c5 f9 6e c6 c4 41 31 ef c9 c4 e2 7d 78 c0 83 e1 3f 83 f9 20 77 34 <c5> 7e
6f 07 c4 c1 7d 74 c8 c4 c1 35 74 d0 c5 ed eb c9 c5 fd d7 c1
[ 5025.370487] RSP: 002b:00007ffed1c48378 EFLAGS: 00010287
[ 5025.370489] RAX: 0000000000000020 RBX: 0000000000000054 RCX:
0000000000000001
[ 5025.370489] RDX: 0000000000000000 RSI: 0000000000000020 RDI:
00007fb1d662a8c1
[ 5025.370490] RBP: 0000000000000001 R08: 0000000000000001 R09:
000056335d747170
[ 5025.370491] R10: 000056335d43c010 R11: 0000000000000202 R12:
000056335d737461
[ 5025.370491] R13: 00007fb1d662a8c1 R14: 0000000000000001 R15:
000056335d737462
[ 5025.370493] Mem-Info:
[ 5025.370497] active_anon:1682354 inactive_anon:269 isolated_anon:0
active_file:35 inactive_file:18 isolated_file:0
unevictable:6680 dirty:7 writeback:1
slab_reclaimable:15094 slab_unreclaimable:22247
mapped:2280 shmem:635 pagetables:5896 bounce:0
free:26442 free_pcp:547 free_cma:0
[ 5025.370500] Node 0 active_anon:6729416kB inactive_anon:1076kB
active_file:140kB inactive_file:72kB unevictable:26720kB isolated(anon):0kB
isolated(file):0kB mapped:9120kB dirty:28kB writeback:4kB shmem:2540kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 65536kB writeback_tmp:0kB
all_unreclaimable? yes
[ 5025.370501] Node 0 DMA free:14884kB min:148kB low:184kB high:220kB
reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB
managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB
free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 5025.370503] lowmem_reserve[]: 0 943 6930 6930 6930
[ 5025.370505] Node 0 DMA32 free:33044kB min:9172kB low:11464kB high:13756kB
reserved_highatomic:0KB active_anon:921492kB inactive_anon:0kB active_file:52kB
inactive_file:192kB unevictable:0kB writepending:8kB present:1032128kB
managed:966592kB mlocked:0kB kernel_stack:1808kB pagetables:1960kB bounce:0kB
free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 5025.370508] lowmem_reserve[]: 0 0 5986 5986 5986
[ 5025.370509] Node 0 Normal free:57840kB min:58256kB low:72820kB
high:87384kB reserved_highatomic:0KB active_anon:5807924kB inactive_anon:1076kB
active_file:296kB inactive_file:0kB unevictable:26720kB writepending:24kB
present:6291456kB managed:6138300kB mlocked:26720kB kernel_stack:21472kB
pagetables:21624kB bounce:0kB free_pcp:2188kB local_pcp:1260kB free_cma:0kB
[ 5025.370512] lowmem_reserve[]: 0 0 0 0 0
[ 5025.370514] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U)
1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14884kB
[ 5025.370519] Node 0 DMA32: 123*4kB (UME) 157*8kB (UME) 91*16kB (UME)
76*32kB (UME) 98*64kB (UME) 75*128kB (UME) 38*256kB (UME) 4*512kB (UME)
0*1024kB 0*2048kB 0*4096kB = 33284kB
[ 5025.370524] Node 0 Normal: 2631*4kB (UME) 1857*8kB (UME) 578*16kB (UME)
167*32kB (UE) 126*64kB (ME) 51*128kB (ME) 15*256kB (ME) 0*512kB 0*1024kB
0*2048kB 0*4096kB = 58404kB
[ 5025.370531] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=1048576kB
[ 5025.370532] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=2048kB
[ 5025.370532] 2746 total pagecache pages
[ 5025.370534] 0 pages in swap cache
[ 5025.370535] Swap cache stats: add 0, delete 0, find 0/0
[ 5025.370535] Free swap = 0kB
[ 5025.370536] Total swap = 0kB
[ 5025.370536] 1834894 pages RAM
[ 5025.370537] 0 pages HighMem/MovableOnly
[ 5025.370537] 54694 pages reserved
[ 5025.370538] 0 pages hwpoisoned
[ 5025.370538] Tasks state (memory values in pages):
[ 5025.370539] [ pid ] uid tgid total_vm rss pgtables_bytes
swapents oom_score_adj name
[ 5025.370543] [ 189] 0 189 19546 1862 126976 0
-250 systemd-journal
[ 5025.370545] [ 223] 0 223 4711 652 57344 0
-1000 systemd-udevd
[ 5025.370547] [ 298] 0 298 1033 670 49152 0
0 hv_kvp_daemon
[ 5025.370548] [ 320] 0 320 269 1 28672 0
0 none
[ 5025.370550] [ 396] 0 396 72100 6492 118784 0
-1000 multipathd
[ 5025.370551] [ 451] 0 451 2076 1512 53248 0
0 haveged
[ 5025.370553] [ 531] 100 531 8734 804 77824 0
0 systemd-network
[ 5025.370554] [ 533] 101 533 5975 1544 81920 0
0 systemd-resolve
[ 5025.370556] [ 647] 0 647 66405 706 102400 0
0 accounts-daemon
[ 5025.370557] [ 655] 0 655 2136 582 53248 0
0 cron
[ 5025.370559] [ 657] 103 657 1920 638 53248 0
-900 dbus-daemon
[ 5025.370560] [ 665] 0 665 22507 512 61440 0
0 irqbalance
[ 5025.370561] [ 666] 112 666 1207 397 53248 0
0 chronyd
[ 5025.370563] [ 673] 112 673 1174 48 53248 0
0 chronyd
[ 5025.370564] [ 674] 0 674 7422 2617 90112 0
0 networkd-dispat
[ 5025.370566] [ 679] 104 679 58175 658 86016 0
0 rsyslogd
[ 5025.370568] [ 680] 0 680 926103 13108 520192 0
0 provisioner
[ 5025.370569] [ 683] 0 683 237560 5885 258048 0
-900 snapd
[ 5025.370571] [ 698] 0 698 4153 672 73728 0
0 systemd-logind
[ 5025.370572] [ 701] 0 701 950 529 49152 0
0 atd
[ 5025.370574] [ 704] 0 704 393832 6976 307200 0
-999 containerd
[ 5025.370575] [ 730] 0 730 1840 404 49152 0
0 agetty
[ 5025.370577] [ 733] 0 733 1459 413 49152 0
0 agetty
[ 5025.370579] [ 735] 0 735 3046 806 61440 0
-1000 sshd
[ 5025.370583] [ 795] 0 795 65252 782 94208 0
0 polkitd
[ 5025.370585] [ 868] 0 868 380845 16531 577536 0
-500 dockerd
[ 5025.370586] [ 1951] 0 1951 341178 2770 188416 0
0 provjobd
[ 5025.370588] [ 1960] 1001 1960 818621 11663 520192 0
0 Runner.Listener
[ 5025.370589] [ 1979] 1001 1979 842838 16000 565248 0
0 Runner.Worker
[ 5025.370591] [ 6469] 0 6469 29756 302 61440 0
-500 unpigz
[ 5025.370592] [ 10928] 0 10928 29756 301 65536 0
-500 unpigz
[ 5025.370594] [ 15766] 0 15766 13372 238 53248 0
-500 unpigz
[ 5025.370596] [ 25231] 1001 25231 66869 2411 606208 0
500 node
[ 5025.370597] [ 25318] 1001 25318 2498 1135 65536 0
500 tmux: server
[ 5025.370599] [ 25319] 1001 25319 179669 1481 122880 0
500 upterm
[ 5025.370600] [ 25330] 1001 25330 1791 488 57344 0
500 tmux: client
[ 5025.370601] [ 25334] 1001 25334 3530 1977 69632 0
500 bash
[ 5025.370603] [ 27869] 1001 27869 1790 546 53248 0
500 tmux: client
[ 5025.370604] [ 28297] 1001 28297 3530 1912 65536 0
500 bash
[ 5025.370606] [ 28463] 1001 28463 2190 533 53248 0
500 bash
[ 5025.370607] [ 28470] 1001 28470 2190 310 49152 0
500 bash
[ 5025.370609] [ 28471] 1001 28471 620550 147455 1589248 0
500 java
[ 5025.370610] [ 28541] 1001 28541 655 124 40960 0
500 sh
[ 5025.370611] [ 28542] 1001 28542 707158 112008 1351680 0
500 java
[ 5025.370613] [ 28619] 0 28619 128871 209 110592 0
-500 docker-proxy
[ 5025.370614] [ 28624] 0 28624 190314 189 143360 0
-500 docker-proxy
[ 5025.370616] [ 28641] 0 28641 177977 433 102400 0
-998 containerd-shim
[ 5025.370617] [ 28662] 0 28662 26831 577 73728 0
0 app
[ 5025.370619] [ 28774] 0 28774 149704 386 126976 0
-500 docker-proxy
[ 5025.370620] [ 28781] 0 28781 190314 185 143360 0
-500 docker-proxy
[ 5025.370622] [ 28801] 0 28801 178041 497 106496 0
-998 containerd-shim
[ 5025.370623] [ 28823] 0 28823 6726 3320 98304 0
0 supervisord
[ 5025.370625] [ 28952] 0 28952 444653 34183 638976 0
0 java
[ 5025.370627] [ 29131] 0 29131 169897 143 135168 0
-500 docker-proxy
[ 5025.370628] [ 29137] 0 29137 149352 225 126976 0
-500 docker-proxy
[ 5025.370629] [ 29154] 0 29154 178329 472 102400 0
-998 containerd-shim
[ 5025.370631] [ 29173] 0 29173 6727 3323 94208 0
0 supervisord
[ 5025.370632] [ 29269] 0 29269 441873 74192 1024000 0
0 java
[ 5025.370633] [ 29655] 0 29655 169897 108 135168 0
-500 docker-proxy
[ 5025.370634] [ 29662] 0 29662 149352 156 126976 0
-500 docker-proxy
[ 5025.370636] [ 29677] 0 29677 178265 581 106496 0
-998 containerd-shim
[ 5025.370637] [ 29696] 0 29696 6729 3334 94208 0
0 supervisord
[ 5025.370639] [ 29813] 0 29813 669683 283833 2748416 0
0 java
[ 5025.370640] [ 29935] 0 29935 149352 106 131072 0
-500 docker-proxy
[ 5025.370641] [ 29941] 0 29941 128871 140 114688 0
-500 docker-proxy
[ 5025.370643] [ 29956] 0 29956 178041 559 110592 0
-998 containerd-shim
[ 5025.370644] [ 29976] 0 29976 6727 3323 94208 0
0 supervisord
[ 5025.370645] [ 30214] 0 30214 672291 285040 2756608 0
0 java
[ 5025.370647] [ 30376] 0 30376 128935 269 118784 0
-500 docker-proxy
[ 5025.370648] [ 30381] 0 30381 108390 172 106496 0
-500 docker-proxy
[ 5025.370649] [ 30393] 0 30393 128935 232 110592 0
-500 docker-proxy
[ 5025.370650] [ 30398] 0 30398 108390 187 114688 0
-500 docker-proxy
[ 5025.370652] [ 30411] 0 30411 177977 634 106496 0
-998 containerd-shim
[ 5025.370653] [ 30430] 0 30430 6732 3344 94208 0
0 supervisord
[ 5025.370654] [ 30643] 0 30643 487711 123392 1437696 0
0 java
[ 5025.370656] [ 31137] 0 31137 1021 26 49152 0
0 tail
[ 5025.370657] [ 31160] 0 31160 149416 232 118784 0
-500 docker-proxy
[ 5025.370658] [ 31166] 0 31166 108390 275 110592 0
-500 docker-proxy
[ 5025.370660] [ 31178] 0 31178 149704 216 122880 0
-500 docker-proxy
[ 5025.370661] [ 31184] 0 31184 149352 233 122880 0
-500 docker-proxy
[ 5025.370663] [ 31200] 0 31200 177977 611 106496 0
-998 containerd-shim
[ 5025.370664] [ 31221] 0 31221 6727 3322 86016 0
0 supervisord
[ 5025.370666] [ 31337] 0 31337 455819 88946 1142784 0
0 java
[ 5025.370667] [ 31658] 0 31658 128935 248 122880 0
-500 docker-proxy
[ 5025.370668] [ 31663] 0 31663 149352 229 118784 0
-500 docker-proxy
[ 5025.370670] [ 31678] 0 31678 178329 641 110592 0
-998 containerd-shim
[ 5025.370671] [ 31698] 0 31698 6729 3324 94208 0
0 supervisord
[ 5025.370673] [ 31814] 0 31814 1084216 358264 3293184 0
0 pulsar-presto-d
[ 5025.370674] [ 32118] 0 32118 1021 26 49152 0
0 tail
[ 5025.370676] [ 32143] 0 32143 1021 26 49152 0
0 tail
[ 5025.370677] [ 32159] 0 32159 1021 26 45056 0
0 tail
[ 5025.370678] [ 35292] 0 35292 2208 599 53248 0
0 jobkeepalive
[ 5025.370680] [ 35879] 1001 35879 74662 6310 241664 0
500 docker
[ 5025.370681] [ 35897] 0 35897 860285 22171 487424 0
0 java
[ 5025.370682] [ 35952] 0 35952 178169 486 106496 0
-998 containerd-shim
[ 5025.370684] [ 35993] 0 35993 1439 125 53248 0
0 bash
[ 5025.370685] [ 36051] 1001 36051 178965 175 110592 0
500 upterm
[ 5025.370687] [ 36066] 1001 36066 655 120 49152 0
500 sh
[ 5025.370688] [ 36067] 1001 36067 2637 635 57344 0
500 ps
[ 5025.370690] [ 36068] 0 36068 25924 4646 139264 0
-500 exe
[ 5025.370691]
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/runner-provisioner.service,task=java,pid=28471,uid=1001
[ 5025.370756] Out of memory: Killed process 28471 (java)
total-vm:2482200kB, anon-rss:589820kB, file-rss:0kB, shmem-rss:0kB, UID:1001
pgtables:1552kB oom_score_adj:500
[ 5025.424552] oom_reaper: reaped process 28471 (java), now anon-rss:0kB,
file-rss:0kB, shmem-rss:0kB
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
