Hello all, Follow two reports from Arnold and Charles (CC'd), it seems vcs0 was experiencing some memory issues earlier today.
from a cursory look: root@vcs0:~# dmesg | grep invoked [833464.613028] sv_aliases invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0 [836237.471697] git invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [838782.826551] git invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [842431.659813] kthreadd invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0 [842431.879033] git invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [852216.818076] cvs2 invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0 The times correspond to 833464 =~ 2017-04-20 03:50 EDT 852216 =~ 2017-04-20 09:02 EDT The processes that were killed each time: root@vcs0:~# dmesg | grep Killed [833464.614349] Killed process 1215 (serve-branches) total-vm:1771440kB, anon-rss:245816kB, file-rss:1132kB [836237.472759] Killed process 8312 (git) total-vm:1125296kB, anon-rss:388512kB, file-rss:660kB [838782.828424] Killed process 8382 (git) total-vm:1126324kB, anon-rss:473972kB, file-rss:708kB [842431.660779] Killed process 8564 (git) total-vm:1125296kB, anon-rss:302448kB, file-rss:876kB [842431.880368] Killed process 8566 (git) total-vm:1125296kB, anon-rss:302456kB, file-rss:996kB [852216.819263] Killed process 12019 (git) total-vm:1125560kB, anon-rss:406192kB, file-rss:564kB Attached is a snippet from dmesg to show the load on vcs0, you could see there many active processes: 46 git-deamon, 51 git-upload-pack, 33 cvs, 15 sshd, etc. For now I believe the temporary load has resolved it self. But in the long run, we'll need more RAM (or add more limits on processes' memory?) regards, - assaf
vcs0-dmesg-2017-04-20.txt.gz
Description: application/gzip
