Hi, A new strange thing, I don't know if you encountered that, today at 6h32 packagekit run, tried to restart gitlab, and failed:
Nov 27 06:32:18 weboob dbus-daemon[464]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.10456' (uid=0 pid=28562 comm="/usr/bin/gdbus call --system --dest org.freedeskto") Nov 27 06:32:18 weboob systemd[1]: Starting PackageKit Daemon... Nov 27 06:32:18 weboob PackageKit: daemon start Nov 27 06:32:18 weboob dbus-daemon[464]: [system] Successfully activated service 'org.freedesktop.PackageKit' Nov 27 06:32:18 weboob systemd[1]: Started PackageKit Daemon. Nov 27 06:32:18 weboob gitlab-workhorse[20628]: git.weboob.org 46.229.168.139 - - [2019/11/27:06:32:18 +0100] "GET /P4ncake/devel/tree/6f5f41fe574ad35b11e024b7c10bb2950d0689b2 HTTP/1.1" 200 40794 "" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" %!f(int64=108) Nov 27 06:32:19 weboob systemd[1]: Reloading. Nov 27 06:32:19 weboob systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly. Nov 27 06:32:19 weboob systemd[1]: Configuration file /etc/systemd/system/postfix.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Nov 27 06:32:19 weboob systemd[1]: /lib/systemd/system/gitlab-sidekiq.service:18: Ignoring unknown escape sequences: "for i in 4 4 4 4 4 4 4 4; do sleep $i; (ps -h -o command -p $MAINPID | grep -q -P "sidekiq \d\.\d\.\d") && exit 0; done" Nov 27 06:32:19 weboob systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly. Nov 27 06:32:20 weboob systemd[1]: Reloading. Nov 27 06:32:20 weboob systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly. Nov 27 06:32:20 weboob systemd[1]: Configuration file /etc/systemd/system/postfix.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Nov 27 06:32:20 weboob systemd[1]: /lib/systemd/system/gitlab-sidekiq.service:18: Ignoring unknown escape sequences: "for i in 4 4 4 4 4 4 4 4; do sleep $i; (ps -h -o command -p $MAINPID | grep -q -P "sidekiq \d\.\d\.\d") && exit 0; done" Nov 27 06:32:20 weboob systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly. Nov 27 06:32:20 weboob systemd[1]: Reloading. Nov 27 06:32:20 weboob systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly. Nov 27 06:32:20 weboob systemd[1]: Configuration file /etc/systemd/system/postfix.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Nov 27 06:32:20 weboob systemd[1]: /lib/systemd/system/gitlab-sidekiq.service:18: Ignoring unknown escape sequences: "for i in 4 4 4 4 4 4 4 4; do sleep $i; (ps -h -o command -p $MAINPID | grep -q -P "sidekiq \d\.\d\.\d") && exit 0; done" Nov 27 06:32:20 weboob systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly. Nov 27 06:32:20 weboob gitaly[20604]: time="2019-11-27T06:32:20+01:00" level=error msg="lstat /var/lib/gitlab/repositories/+gitaly/cache: no such file or directory" storage=default Nov 27 06:32:20 weboob systemd[1]: Reloading. Nov 27 06:32:21 weboob systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly. Nov 27 06:32:21 weboob systemd[1]: Configuration file /etc/systemd/system/postfix.service.d/override.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway. Nov 27 06:32:21 weboob systemd[1]: /lib/systemd/system/gitlab-sidekiq.service:18: Ignoring unknown escape sequences: "for i in 4 4 4 4 4 4 4 4; do sleep $i; (ps -h -o command -p $MAINPID | grep -q -P "sidekiq \d\.\d\.\d") && exit 0; done" Nov 27 06:32:21 weboob systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly. Nov 27 06:32:21 weboob systemd[1]: Stopped target GitLab Service. Nov 27 06:32:21 weboob systemd[1]: Stopping GitLab Service. Nov 27 06:32:21 weboob systemd[1]: gitlab.service: Succeeded. Nov 27 06:32:21 weboob systemd[1]: Stopped GitLab Services. Nov 27 06:32:21 weboob systemd[1]: Stopping GitLab Services... Nov 27 06:32:21 weboob systemd[1]: Stopping Gitlab mailroom Worker... Nov 27 06:32:21 weboob systemd[1]: Stopping GitLab Sidekiq Worker... Nov 27 06:32:21 weboob systemd[1]: Stopping Gitlab Workhorse handles slow HTTP requests for Gitlab.... Nov 27 06:32:21 weboob gitlab-sidekiq[20644]: 2019-11-27T05:32:21.133Z 20644 TID-goijjrii4 INFO: Shutting down Nov 27 06:32:21 weboob gitlab-sidekiq[20644]: 2019-11-27T05:32:21.133Z 20644 TID-goin08v3c INFO: Scheduler exiting... Nov 27 06:32:21 weboob gitlab-sidekiq[20644]: 2019-11-27T05:32:21.133Z 20644 TID-goijjrii4 INFO: Terminating quiet workers Nov 27 06:32:21 weboob gitlab-sidekiq[20644]: 2019-11-27T05:32:21.135Z 20644 TID-goin0a0a0 INFO: Scheduler exiting... Nov 27 06:32:21 weboob systemd[1]: gitlab-workhorse.service: Succeeded. Nov 27 06:32:21 weboob systemd[1]: Stopped Gitlab Workhorse handles slow HTTP requests for Gitlab.. Nov 27 06:32:21 weboob systemd[1]: gitlab-mailroom.service: Succeeded. Nov 27 06:32:21 weboob systemd[1]: Stopped Gitlab mailroom Worker. Nov 27 06:32:21 weboob systemd[1]: Stopping GitLab Unicorn Server... Nov 27 06:32:21 weboob systemd[1]: gitlab-unicorn.service: Succeeded. Nov 27 06:32:21 weboob systemd[1]: Stopped GitLab Unicorn Server. Nov 27 06:32:21 weboob gitlab-sidekiq[20644]: 2019-11-27T05:32:21.635Z 20644 TID-goijjrii4 INFO: Pausing to allow workers to finish... Nov 27 06:32:23 weboob gitlab-sidekiq[20644]: 2019-11-27T05:32:23.639Z 20644 TID-goijjrii4 INFO: Bye! Nov 27 06:32:23 weboob systemd[1]: gitlab-sidekiq.service: Succeeded. Nov 27 06:32:23 weboob systemd[1]: Stopped GitLab Sidekiq Worker. Nov 27 06:32:23 weboob systemd[1]: Stopping Advanced key-value store... Nov 27 06:32:24 weboob systemd[1]: redis-server.service: Succeeded. Nov 27 06:32:24 weboob systemd[1]: Stopped Advanced key-value store. Nov 27 06:32:24 weboob systemd[1]: Starting Advanced key-value store... Nov 27 06:32:24 weboob systemd[1]: redis-server.service: Can't open PID file /run/redis/redis-server.pid (yet?) after start: No such file or directory Nov 27 06:32:24 weboob systemd[1]: Started Advanced key-value store. Nov 27 06:32:24 weboob systemd[1]: Starting GitLab Sidekiq Worker... Nov 27 06:32:24 weboob systemd[1]: gitlab-unicorn.service: Ignoring invalid environment assignment 'for i in $(grep -v '#' /etc/gitlab/gitlab-debian.conf | cut -d=-f 1)': /etc/default/gitlab Nov 27 06:32:24 weboob systemd[1]: gitlab-unicorn.service: Ignoring invalid environment assignment 'export app_user=${gitlab_user}': /etc/default/gitlab Nov 27 06:32:24 weboob systemd[1]: Started GitLab Unicorn Server. Nov 27 06:32:24 weboob systemd[1]: Started Gitlab mailroom Worker. Nov 27 06:32:24 weboob systemd[1]: Started Gitlab Workhorse handles slow HTTP requests for Gitlab.. Nov 27 06:32:24 weboob gitlab-workhorse[28833]: time="2019-11-27T06:32:24+01:00" level=info msg=Starting version="gitlab-workhorse (unknown version)" Nov 27 06:32:24 weboob gitlab-unicorn[28831]: fatal: not a git repository (or any of the parent directories): .git Nov 27 06:32:24 weboob gitlab-unicorn[28831]: fatal: not a git repository (or any of the parent directories): .git Nov 27 06:32:24 weboob gitlab-unicorn[28831]: bundler: failed to load command: unicorn_rails (/usr/local/bin/unicorn_rails) Nov 27 06:32:24 weboob gitlab-unicorn[28831]: Bundler::GemNotFound: Could not find gem 'rubyzip (>= 1.2.2, ~> 1.2)' in any of the gem sources listed in your Gemfile. Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:287:in `block in verify_gemfile_dependencies_are_found!' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:255:in `each' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:255:in `verify_gemfile_dependencies_are_found!' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:49:in `start' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:22:in `resolve' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/definition.rb:258:in `resolve' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/definition.rb:170:in `specs' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/definition.rb:237:in `specs_for' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/definition.rb:226:in `requested_specs' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/runtime.rb:108:in `block in definition_method' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/runtime.rb:20:in `setup' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler.rb:107:in `setup' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/setup.rb:20:in `<top (required)>' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' Nov 27 06:32:24 weboob gitlab-unicorn[28831]: /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' Nov 27 06:32:24 weboob systemd[1]: gitlab-unicorn.service: Main process exited, code=exited, status=1/FAILURE Nov 27 06:32:24 weboob systemd[1]: gitlab-unicorn.service: Failed with result 'exit-code'. Nov 27 06:32:24 weboob gitlab-mailroom[28832]: fatal: not a git repository (or any of the parent directories): .git Nov 27 06:32:24 weboob gitlab-mailroom[28832]: fatal: not a git repository (or any of the parent directories): .git Nov 27 06:32:24 weboob gitlab-sidekiq[28829]: fatal: not a git repository (or any of the parent directories): .git Nov 27 06:32:24 weboob gitlab-sidekiq[28829]: fatal: not a git repository (or any of the parent directories): .git Nov 27 06:32:24 weboob gitlab-mailroom[28832]: /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:287:in `block in verify_gemfile_dependencies_are_found!': Could not find gem 'rubyzip (>= 1.2.2, ~> 1.2)' in any of the gem sources listed in your Gemfile. (Bundler::GemNotFound) Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:255:in `each' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:255:in `verify_gemfile_dependencies_are_found!' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:49:in `start' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/resolver.rb:22:in `resolve' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/definition.rb:258:in `resolve' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/definition.rb:170:in `specs' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/definition.rb:237:in `specs_for' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/definition.rb:226:in `requested_specs' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/runtime.rb:108:in `block in definition_method' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/runtime.rb:20:in `setup' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler.rb:107:in `setup' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/share/rubygems-integration/all/gems/bundler-1.17.3/lib/bundler/setup.rb:20:in `<top (required)>' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' Nov 27 06:32:24 weboob gitlab-mailroom[28832]: #011from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' Nov 27 06:32:24 weboob systemd[1]: gitlab-mailroom.service: Main process exited, code=exited, status=1/FAILURE Nov 27 06:32:24 weboob systemd[1]: gitlab-mailroom.service: Failed with result 'exit-code'. I had to install manually rubyzip with gem at the right version to fix gitlab. To understand, packagekit has updated something? Romain
