Package: gitlab
Severity: important
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hello,
we have low mem instance of gitlab - it used to have 2GB of ram, now it has 3GB.
Sometimes, it runs out of memory. (usually at 22:xx )
OOM killer usually kills one of sidekiq children.
It triggers restart of sidekiq, but also of other gitlab services.
But gitlab-sidekiq is the only one, which is started, other are left dead.
Can this be somehow addressed in systemd service dependecies?
Libor
-----BEGIN PGP SIGNATURE-----
iQJJBAEBCAAzFiEEPGZVVU37tFmB0TQv8O+MbsKfR44FAliLGIQVHGxpYm9yLmts
ZXBhY0BiY29tLmN6AAoJEPDvjG7Cn0eOcGgP/3eJAMqQoBWxG48YXLnfx1D6YZ6h
K4CLe0fpz8rFG87v4qsfyszA5WF3qY3QTffYKXxDJq5Z9QAaV2V+k5Jn0j0Sqhes
IsZqN5g4XvOdyYB6UEy6IGdDhiogiqCeHTz/gRL83WYDb2c+aIykZVYUWSS2MELl
fIpDyIn5FvlRs42VQu0u6x/Wi6b1+j55Y7KWo3j/bET+8MQzg5fOEdIO3qXGTGrf
A5wM0QQtFsOmcvQNWz7hWn7Lm4Fd5Q5U48O0MRiaJNTOUA/OOfoHgRBwHcBAUtvR
2alVQwb9FlH90wdRhUHARJEX10m2+duTvRw4KrP2PHaFcy5l/GJiVp8RV5NrAkHF
Fo9xDrmNjDnCAhjT54gOaudAYW5P9MpqJDHOtcaenvsoK8OHaEFh85kBxYhDj4DA
T8f7qnoHzBsHXzDr3zLMas8TGVfWRc/H7HhhL19WTDwCyXJfm+NBcwYd5RUXysl/
TdZE9K09uhxKtMNbnhAqfPk08XTmTRLObRGx/2U8wWaU8f3X1aZOJLANOQUKuq2Q
57eEIQ3Ed8eY8AUN95jfxq5s9QbgLhwGf8hl6eyzXKrAtHxq/coeN2rNzdaEYIac
A7VRgk4id1qnwqdAFgy6DoNTzqK2vcAjy5Ix+DmoFPHmaxqIuXj6O9pNr5+/9lL4
/0DxTlrb5zieOvBi
=h2TC
-----END PGP SIGNATURE-----
Jan 26 22:15:18 gitlab kernel: kworker/0:0 invoked oom-killer:
gfp_mask=0x24200c2(GFP_HIGHUSER), order=0, oom_score_adj=0
Jan 26 22:15:18 gitlab kernel: kworker/0:0 cpuset=/ mems_allowed=0
Jan 26 22:15:18 gitlab kernel: CPU: 0 PID: 9360 Comm: kworker/0:0 Not tainted
4.8.0-2-amd64 #1 Debian 4.8.15-2
Jan 26 22:15:18 gitlab kernel: Hardware name: VMware, Inc. VMware Virtual
Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
Jan 26 22:15:18 gitlab kernel: Workqueue: events_freezable vmballoon_work
[vmw_balloon]
Jan 26 22:15:18 gitlab kernel: 0000000000000286 000000001c838855
ffffffff8831f925 ffff972ef1ab3d08
Jan 26 22:15:18 gitlab kernel: ffff972f38b02080 ffffffff881f7491
0000000000000206 ffffffff8818f2e9
Jan 26 22:15:18 gitlab kernel: ffff972f3fffccc0 0000000c00000246
ffff972f2d66c080 ffff972f38b02080
Jan 26 22:15:18 gitlab kernel: Call Trace:
Jan 26 22:15:18 gitlab kernel: [<ffffffff8831f925>] ? dump_stack+0x5c/0x77
Jan 26 22:15:18 gitlab kernel: [<ffffffff881f7491>] ? dump_header+0x59/0x1dc
Jan 26 22:15:18 gitlab kernel: [<ffffffff8818f2e9>] ?
do_try_to_free_pages+0x269/0x350
Jan 26 22:15:18 gitlab kernel: [<ffffffff8817a392>] ?
oom_kill_process+0x222/0x3e0
Jan 26 22:15:18 gitlab kernel: [<ffffffff8817a8fc>] ? out_of_memory+0x35c/0x3f0
Jan 26 22:15:18 gitlab kernel: [<ffffffff8817f987>] ?
__alloc_pages_nodemask+0xd67/0xda0
Jan 26 22:15:18 gitlab kernel: [<ffffffff880a7101>] ? update_curr+0xe1/0x160
Jan 26 22:15:18 gitlab kernel: [<ffffffff881d0221>] ?
alloc_pages_current+0x91/0x140
Jan 26 22:15:18 gitlab kernel: [<ffffffffc04fcf5f>] ?
vmballoon_work+0x47f/0x704 [vmw_balloon]
Jan 26 22:15:18 gitlab kernel: [<ffffffff8808dc90>] ?
process_one_work+0x160/0x410
Jan 26 22:15:18 gitlab kernel: [<ffffffff8808df8d>] ? worker_thread+0x4d/0x480
Jan 26 22:15:18 gitlab kernel: [<ffffffff8808df40>] ?
process_one_work+0x410/0x410
Jan 26 22:15:18 gitlab kernel: [<ffffffff88093d5d>] ? kthread+0xcd/0xf0
Jan 26 22:15:18 gitlab kernel: [<ffffffff885e8eef>] ? ret_from_fork+0x1f/0x40
Jan 26 22:15:18 gitlab kernel: [<ffffffff88093c90>] ?
kthread_create_on_node+0x190/0x190
Jan 26 22:15:18 gitlab kernel: Mem-Info:
Jan 26 22:15:18 gitlab kernel: active_anon:98986 inactive_anon:102318
isolated_anon:0
active_file:23 inactive_file:47
isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
slab_reclaimable:3358 slab_unreclaimable:9520
mapped:1580 shmem:1984 pagetables:3919
bounce:0
free:14226 free_pcp:30 free_cma:0
Jan 26 22:15:18 gitlab kernel: Node 0 active_anon:395944kB
inactive_anon:409272kB active_file:92kB inactive_file:188kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB mapped:6320kB dirty:0kB writeback:0kB
shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 7936kB
writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Jan 26 22:15:18 gitlab kernel: Node 0 DMA free:12136kB min:232kB low:288kB
high:344kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB
bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 26 22:15:18 gitlab kernel: lowmem_reserve[]: 0 2976 2976 2976 2976
Jan 26 22:15:18 gitlab kernel: Node 0 DMA32 free:44768kB min:44820kB
low:56024kB high:67228kB active_anon:395944kB inactive_anon:409272kB
active_file:92kB inactive_file:188kB unevictable:0kB writepending:0kB
present:3129280kB managed:3068696kB mlocked:0kB slab_reclaimable:13432kB
slab_unreclaimable:38080kB kernel_stack:3648kB pagetables:15676kB bounce:0kB
free_pcp:120kB local_pcp:120kB free_cma:0kB
Jan 26 22:15:18 gitlab kernel: lowmem_reserve[]: 0 0 0 0 0
Jan 26 22:15:18 gitlab kernel: Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U)
1*64kB (U) 0*128kB 1*256kB (U) 1*512kB (U) 1*1024kB (U) 1*2048kB (M) 2*4096kB
(ME) = 12136kB
Jan 26 22:15:18 gitlab kernel: Node 0 DMA32: 10060*4kB (UME) 576*8kB (UM)
0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
44848kB
Jan 26 22:15:18 gitlab kernel: Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
Jan 26 22:15:18 gitlab kernel: 12168 total pagecache pages
Jan 26 22:15:18 gitlab kernel: 10120 pages in swap cache
Jan 26 22:15:18 gitlab kernel: Swap cache stats: add 987900, delete 977780,
find 2147165/2439256
Jan 26 22:15:18 gitlab kernel: Free swap = 0kB
Jan 26 22:15:18 gitlab kernel: Total swap = 999420kB
Jan 26 22:15:18 gitlab kernel: 786318 pages RAM
Jan 26 22:15:18 gitlab kernel: 0 pages HighMem/MovableOnly
Jan 26 22:15:18 gitlab kernel: 15167 pages reserved
Jan 26 22:15:18 gitlab kernel: 0 pages hwpoisoned
Jan 26 22:15:18 gitlab kernel: [ pid ] uid tgid total_vm rss nr_ptes
nr_pmds swapents oom_score_adj name
Jan 26 22:15:18 gitlab kernel: [ 253] 0 253 14533 80 24
3 82 0 systemd-journal
Jan 26 22:15:18 gitlab kernel: [ 276] 0 276 26800 0 19
4 85 0 lvmetad
Jan 26 22:15:18 gitlab kernel: [ 280] 0 280 11688 12 24
4 503 -1000 systemd-udevd
Jan 26 22:15:18 gitlab kernel: [ 475] 0 475 11639 32 27
3 129 0 systemd-logind
Jan 26 22:15:18 gitlab kernel: [ 476] 0 476 6991 0 19
4 54 0 atd
Jan 26 22:15:18 gitlab kernel: [ 477] 0 477 62528 62 29
4 226 0 rsyslogd
Jan 26 22:15:18 gitlab kernel: [ 482] 105 482 11278 50 26
3 67 -900 dbus-daemon
Jan 26 22:15:18 gitlab kernel: [ 513] 0 513 6997 32 18
3 46 0 cron
Jan 26 22:15:18 gitlab kernel: [ 530] 0 530 51506 102 57
3 258 0 vmtoolsd
Jan 26 22:15:18 gitlab kernel: [ 537] 0 537 1054 0 8
3 30 0 acpid
Jan 26 22:15:18 gitlab kernel: [ 719] 108 719 5949 14 18
3 90 -500 nrpe
Jan 26 22:15:18 gitlab kernel: [ 728] 0 728 17489 11 39
3 183 -1000 sshd
Jan 26 22:15:18 gitlab kernel: [ 737] 0 737 3662 0 11
3 34 0 agetty
Jan 26 22:15:18 gitlab kernel: [ 756] 110 756 25549 70 26
3 108 0 ntpd
Jan 26 22:15:18 gitlab kernel: [ 765] 0 765 12592 105 32
4 2247 0 munin-node
Jan 26 22:15:18 gitlab kernel: [ 818] 114 818 10828 365 19
3 1130 0 redis-server
Jan 26 22:15:18 gitlab kernel: [ 831] 113 831 68188 329 57
3 369 -900 postgres
Jan 26 22:15:18 gitlab kernel: [ 832] 113 832 72342 376 67
3 402 -900 postgres
Jan 26 22:15:18 gitlab kernel: [ 841] 0 841 20287 17 28
3 185 0 master
Jan 26 22:15:18 gitlab kernel: [ 852] 109 852 20845 20 32
3 199 0 qmgr
Jan 26 22:15:18 gitlab kernel: [ 859] 0 859 24571 26 53
3 453 0 apache2
Jan 26 22:15:18 gitlab kernel: [ 919] 33 919 5393 18 15
3 32 0 htcacheclean
Jan 26 22:15:18 gitlab kernel: [ 922] 113 922 72342 77 54
3 420 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 923] 113 923 72342 82 55
3 414 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 924] 113 924 72342 64 54
3 422 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 925] 113 925 72454 228 57
3 445 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 926] 113 926 36092 47 52
3 412 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 939] 113 939 68221 329 62
3 399 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 940] 113 940 68188 117 50
3 378 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 941] 113 941 68188 61 51
3 388 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 942] 113 942 68282 276 53
3 396 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 943] 113 943 32010 106 46
3 365 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1130] 115 1130 404015 64570 410
6 56881 0 sidekiq
Jan 26 22:15:18 gitlab kernel: [ 1133] 115 1133 160074 7295 269
4 55461 0 unicorn_rails
Jan 26 22:15:18 gitlab kernel: [ 1134] 115 1134 28119 23 58
3 5872 0 mail_room
Jan 26 22:15:18 gitlab kernel: [ 1135] 115 1135 31701 423 21
5 74 0 gitlab-workhors
Jan 26 22:15:18 gitlab kernel: [ 1613] 113 1613 70206 481 59
3 747 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1698] 115 1698 175756 53397 295
4 23918 0 unicorn_rails
Jan 26 22:15:18 gitlab kernel: [ 1701] 115 1701 178468 62348 298
4 16520 0 unicorn_rails
Jan 26 22:15:18 gitlab kernel: [ 1784] 113 1784 104096 4142 135
3 28366 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1793] 113 1793 109673 5080 148
4 33780 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1825] 113 1825 71767 854 71
3 1486 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1826] 113 1826 71698 891 71
3 1367 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1870] 113 1870 72056 853 72
3 1724 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1963] 113 1963 72127 730 72
3 1942 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1975] 113 1975 71909 970 71
3 1503 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1976] 113 1976 72064 759 72
3 1813 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1977] 113 1977 71127 721 70
3 1449 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1978] 113 1978 72159 905 72
3 1775 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1979] 113 1979 72078 929 72
3 1667 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1983] 113 1983 71024 850 70
3 1254 0 postgres
Jan 26 22:15:18 gitlab kernel: [ 1990] 113 1990 71668 831 71
3 1890 0 postgres
Jan 26 22:15:18 gitlab kernel: [30182] 33 30182 326710 450 97
5 1507 0 apache2
Jan 26 22:15:18 gitlab kernel: [30183] 33 30183 326630 477 97
4 1356 0 apache2
Jan 26 22:15:18 gitlab kernel: [18263] 109 18263 20803 194 31
3 0 0 pickup
Jan 26 22:15:18 gitlab kernel: Out of memory: Kill process 1130 (sidekiq) score
119 or sacrifice child
Jan 26 22:15:18 gitlab kernel: Killed process 1130 (sidekiq)
total-vm:1616060kB, anon-rss:258280kB, file-rss:0kB, shmem-rss:0kB
Jan 26 22:15:18 gitlab kernel: oom_reaper: reaped process 1130 (sidekiq), now
anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jan 26 22:15:18 gitlab systemd[1]: gitlab-sidekiq.service: Main process exited,
code=killed, status=9/KILL
Jan 26 22:15:18 gitlab systemd[1]: gitlab-sidekiq.service: Unit entered failed
state.
Jan 26 22:15:18 gitlab systemd[1]: gitlab-sidekiq.service: Failed with result
'signal'.
Jan 26 22:15:18 gitlab systemd[1]: Stopped target GitLab Service.
Jan 26 22:15:18 gitlab systemd[1]: Stopped GitLab Services.
Jan 26 22:15:18 gitlab systemd[1]: Stopping Gitlab mailroom Worker...
Jan 26 22:15:18 gitlab systemd[1]: Stopping Gitlab Workhorse handles slow HTTP
requests for Gitlab....
Jan 26 22:15:18 gitlab systemd[1]: Stopped Gitlab Workhorse handles slow HTTP
requests for Gitlab..
Jan 26 22:15:19 gitlab systemd[1]: Stopped Gitlab mailroom Worker.
Jan 26 22:15:19 gitlab systemd[1]: Stopping GitLab Unicorn Server...
Jan 26 22:15:20 gitlab systemd[1]: Stopped GitLab Unicorn Server.
Jan 26 22:15:28 gitlab systemd[1]: gitlab-sidekiq.service: Service hold-off
time over, scheduling restart.
Jan 26 22:15:28 gitlab systemd[1]: Stopped GitLab Sidekiq Worker.
Jan 26 22:15:28 gitlab systemd[1]: Starting GitLab Sidekiq Worker...
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.243Z 18868
TID-gn57l97t8 INFO: Booting Sidekiq 4.2.3 with redis options
{:host=>"localhost", :port=>6379, :password=>nil, :namespace=>"resque:gitlab",
:url=>nil}
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.295Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name: stuck_ci_builds_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.301Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name: expire_build_artifacts_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.310Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name: repository_check_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.320Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name: admin_email_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.325Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name:
repository_archive_cache_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.330Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name:
import_export_project_cleanup_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.334Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name: requests_profiles_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.339Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name: remove_expired_members_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.344Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name:
remove_expired_group_links_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.347Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name: prune_old_events_worker
Jan 26 22:15:47 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:15:47.354Z 18868
TID-gn57l97t8 INFO: Cron Jobs - add job with name: trending_projects_worker
Jan 26 22:16:01 gitlab systemd[1]: Started GitLab Sidekiq Worker.
Jan 26 22:16:04 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:16:04.347Z 18868
TID-gn57l97t8 INFO: Running in ruby 2.3.3p222 (2016-11-21) [x86_64-linux-gnu]
Jan 26 22:16:04 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:16:04.347Z 18868
TID-gn57l97t8 INFO: See LICENSE and the LGPL-3.0 for licensing details.
Jan 26 22:16:04 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:16:04.347Z 18868
TID-gn57l97t8 INFO: Upgrade to Sidekiq Pro for more features and support:
http://sidekiq.org
Jan 26 22:16:04 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:16:04.354Z 18868
TID-gn57l97t8 INFO: Starting processing, hit Ctrl-C to stop
Jan 26 22:20:05 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:20:05.289Z 18868
TID-gn58yenr0 RepositoryCheck::BatchWorker JID-5d784fb4b3f6f9c8f0381712 INFO:
start
Jan 26 22:20:05 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:20:05.292Z 18868
TID-gn58ydpxs INFO: Cron Jobs - add job with name: repository_check_worker
Jan 26 22:20:05 gitlab gitlab-sidekiq[18868]: 2017-01-26T21:20:05.325Z 18868
TID-gn58yenr0 RepositoryCheck::BatchWorker JID-5d784fb4b3f6f9c8f0381712 INFO:
done: 0.036 sec
