Package: gitlab
Version: 8.4.3+dfsg-6
Severity: grave
Justification: renders package unusable
Hi,
this problem appeared after my upgrade to 8.4.3+dfsg-6 without me
changing anything in my configuration, so I assume that this must be
gitlab's fault, thus the high severity.
The symptom is, that since after the upgrade (and after manually
creating the /var/run/gitlab directories as I described in #814713), I
only get the following message in my browser when trying to access the
website:
502
Whoops, GitLab is taking too much time to respond.
Try refreshing the page, or going back and attempting the action again.
Please contact your GitLab administrator if this problem persists.
Now I *am* the gitlab administrator and I fail to figure out how to find
the source of the problem. My logs show the following:
root@22gitlab:~# tail -n10000 /usr/share/gitlab/log/* /var/log/gitlab/*
==> /usr/share/gitlab/log/mail_room.log <==
fatal: Not a git repository (or any of the parent directories): .git
==> /usr/share/gitlab/log/production.log <==
** [Raven] Raven 0.15.3 configured not to send errors.
** [Raven] Raven 0.15.3 configured not to send errors.
** [Raven] Raven 0.15.3 configured not to send errors.
** [Raven] Raven 0.15.3 configured not to send errors.
Scoped order and limit are ignored, it's forced to be batch order and
batch size
** [Raven] Raven 0.15.3 configured not to send errors.
Scoped order and limit are ignored, it's forced to be batch order and
batch size
==> /var/log/gitlab/sidekiq.log <==
No such file or directory @ rb_sysopen -
/var/run/gitlab/pids/sidekiq.pid
/usr/lib/ruby/vendor_ruby/sidekiq/cli.rb:355:in `initialize'
/usr/lib/ruby/vendor_ruby/sidekiq/cli.rb:355:in `open'
/usr/lib/ruby/vendor_ruby/sidekiq/cli.rb:355:in `write_pid'
/usr/lib/ruby/vendor_ruby/sidekiq/cli.rb:42:in `parse'
/usr/bin/sidekiq:12:in `<main>'
Pidfile /var/run/gitlab/pids/sidekiq.pid does not exist
2016-02-14T15:21:41.835Z 293 TID-b4ak0 INFO: Booting Sidekiq 4.0.1 with
redis options {:url=>"redis://localhost:6379", :namespace=>"resque:gitlab"}
2016-02-14T15:21:41.983Z 293 TID-b4ak0 INFO: Cron Jobs - add job with
name: stuck_ci_builds_worker
fatal: Not a git repository (or any of the parent directories): .git
2016-02-14T15:21:57.436Z 293 TID-b4ak0 INFO: Running in ruby 2.2.4p230
(2015-12-16) [x86_64-linux-gnu]
2016-02-14T15:21:57.436Z 293 TID-b4ak0 INFO: See LICENSE and the
LGPL-3.0 for licensing details.
2016-02-14T15:21:57.436Z 293 TID-b4ak0 INFO: Upgrade to Sidekiq Pro for
more features and support: http://sidekiq.org
==> /var/log/gitlab/unicorn.stderr.log <==
I, [2016-02-14T16:21:07.730554 #287] INFO -- : Refreshing Gem list
fatal: Not a git repository (or any of the parent directories): .git
I, [2016-02-14T16:21:58.169242 #287] INFO -- : listening on
addr=/var/run/gitlab/pids/gitlab.socket fd=12
I, [2016-02-14T16:21:58.170435 #287] INFO -- : listening on
addr=127.0.0.1:8080 fd=15
I, [2016-02-14T16:21:58.195925 #346] INFO -- : worker=0 ready
I, [2016-02-14T16:21:58.205334 #349] INFO -- : worker=1 ready
I, [2016-02-14T16:21:58.220556 #287] INFO -- : master process ready
I, [2016-02-14T16:21:58.223929 #352] INFO -- : worker=2 ready
==> /var/log/gitlab/unicorn.stdout.log <==
Does there exist a way to increase the verbosity of the logging so that
I can see what causes the 502 problem? Here is the (shortened, because I
have lots of repositories) output of gitlab:check:
fatal: Not a git repository (or any of the parent directories): .git
Checking GitLab Shell ...
GitLab Shell version >= 2.6.10 ? ... OK (2.6.10)
Repo base directory exists? ... yes
Repo base directory is a symlink? ... no
Repo base owned by gitlab:gitlab? ... yes
Repo base access is drwxrws---? ... yes
hooks directories in repos are links: ...
[...]
Running /usr/share/gitlab-shell/bin/check
Check GitLab API access: FAILED. code: 502
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u gitlab -H editor /usr/share/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
Database is SQLite ... no
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... skipped (no tmp uploads folder
yet)
Init script exists? ... yes
Init script up-to-date? ... yes
projects have namespace: ...
[...]
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.2.4)
Your git bin path is "/usr/bin/git"
Git version >= 1.7.10 ? ... yes (2.1.4)
Active users: 16
Checking GitLab ... Finished
So this is all green except for the gitlab-shell/bin/check output. But I
guess it's expected that this is red because I cannot access the web
interface through my browser either, right?
I'd appreciate some advise in how to tell you more about the reasons for
this problem.
Thanks!
cheers, josch
