--- Begin Message ---
Package: gitlab
Version: 8.13.11+dfsg1-8+deb9u2
Severity: normal
* What led up to the situation?
I upgraded to the latest security update (8.13.11+dfsg1-8+deb9u2) and
rebooted the box.
* What exactly did you do (or not do) that was effective (or
ineffective)?
I tried creating a new merge request.
* What was the outcome of this action?
Gitlab throws a 500 error "Whoops, something went wrong on our end." The
merge request is indeed not created (it does not show up in the merge
request list, which has other, previously created entries)
/var/log/gitlab/production.log shows the following error:
Processing by ProjectsController#autocomplete_sources as JSON
Parameters: {"type"=>"MergeRequest", "namespace_id"=>"operations",
"id"=>"ems"}
Completed 200 OK in 502ms (Views: 169.8ms | ActiveRecord: 53.2ms)
Started POST "/operations/ems/merge_requests" for [redacted ip address]
at 2018-05-25 16:09:41 +0100
Processing by Projects::MergeRequestsController#create as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[redacted token]",
"merge_request"=>{"title"=>"[redacted merge request title]",
"description"=>"", "label_ids"=>[""], "force_remove_source_branch"=>"0",
"lock_version"=>"0", "source_project_id"=>"1",
"source_branch"=>"[redacted source git branch]",
"target_project_id"=>"1", "target_branch"=>"master"},
"namespace_id"=>"operations", "project_id"=>"ems"}
Completed 500 Internal Server Error in 123ms (ActiveRecord: 13.2ms)
NameError (undefined local variable or method `source_project' for
#<MergeRequests::CreateService:0x0055dc04005920>
Did you mean? @source_project):
app/services/merge_requests/create_service.rb:6:in `execute'
app/controllers/projects/merge_requests_controller.rb:254:in `create'
lib/gitlab/request_profiler/middleware.rb:15:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
* What outcome did you expect instead?
A merge request should have been created just fine. I should have been
taken to the created merge request page instead of being shown an error
page.
Earlier this morning before the upgrade, merge requests could be created
just fine. The system is fully up to date.
I tried re-installing gitlab with apt-get install --reinstall gitlab.
Rake tasks (which I assume were ran by the post-install script)
pre-compiled a bunch of assets once again and validated my config and
projects, but merge requests still can't be created.
Browsing projects/issues/other pages seem to work fine, although I
haven't checked every possible action.
-- System Information:
Debian Release: 9.4
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.9.0-6-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_GB:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages gitlab depends on:
ii adduser 3.115
ii asciidoctor 1.5.4-2
ii bc 1.06.95-9+b3
ii bundler 1.13.6-2
ii dbconfig-pgsql 2.0.8
ii debconf [debconf-2.0] 1.5.61
ii git 1:2.11.0-3+deb9u2
ii gitlab-shell 3.6.6-4
ii gitlab-workhorse 0.8.5+debian-3+b2
ii init-system-helpers 1.48
ii libjs-chartjs 1.0.2-1
ii libjs-clipboard 1.4.2-1
ii libjs-fuzzaldrin-plus 0.3.1+git.20161008.da2cb58+dfsg-4
ii libjs-graphael 0.5+dfsg-1
ii libjs-jquery-cookie 11-3
ii libjs-jquery-history 11-3
ii libjs-jquery-nicescroll 3.6.6-1
ii lsb-base 9.20161125
ii nginx 1.10.3-1+deb9u1
ii nginx-full [nginx] 1.10.3-1+deb9u1
ii nodejs 4.8.2~dfsg-1
ii openssh-client 1:7.4p1-10+deb9u3
ii postfix [mail-transport-agent] 3.1.8-0+deb9u1
ii postgresql-client 9.6+181+deb9u1
ii postgresql-client-9.6 [postgresql-client 9.6.7-0+deb9u1
ii postgresql-contrib 9.6+181+deb9u1
ii rake 10.5.0-2
ii redis-server 3:3.2.6-1
ii ruby 1:2.3.3
ii ruby-ace-rails-ap 4.1.1-1
ii ruby-activerecord-session-store 1.0.0-2
ii ruby-acts-as-taggable-on 4.0.0-2
ii ruby-addressable 2.4.0-1
ii ruby-after-commit-queue 1.3.0-1
ii ruby-akismet 2.0.0-1
ii ruby-allocations 1.0.3-1+b2
ii ruby-asana 0.4.0-1
ii ruby-attr-encrypted 3.0.1-2
ii ruby-babosa 1.0.2-2
ii ruby-base32 0.3.2-3
ii ruby-bootstrap-sass 3.3.5.1-5
ii ruby-browser 2.2.0-2
ii ruby-cal-heatmap-rails 3.6.0+dfsg-1
ii ruby-carrierwave 0.10.0+gh-4
ii ruby-charlock-holmes 0.7.3+dfsg-2+b3
ii ruby-chronic 0.10.2-3
ii ruby-chronic-duration 0.10.6-1
ii ruby-coffee-rails 4.1.0-2
ii ruby-coffee-script-source 1.10.0-1
ii ruby-connection-pool 2.2.0-1
ii ruby-creole 0.5.0-2
ii ruby-d3-rails 3.5.6+dfsg-1
ii ruby-default-value-for 3.0.1-1
ii ruby-devise 4.2.0-1
ii ruby-devise-two-factor 3.0.0-2
ii ruby-diffy 3.0.6-1
ii ruby-doorkeeper 4.2.0-3
ii ruby-dropzonejs-rails 0.7.1-1
ii ruby-email-reply-parser 0.5.8-1
ii ruby-fog-aws 0.12.0-1
ii ruby-fog-azure 0.0.2-1
ii ruby-fog-core 1.42.0-1
ii ruby-fog-google 0.3.2-1
ii ruby-fog-local 0.3.0-1
ii ruby-fog-openstack 0.1.6-4
ii ruby-fog-rackspace 0.1.1-4
ii ruby-fogbugz 0.2.1-3
ii ruby-font-awesome-rails 4.6.3.0-2
ii ruby-gemnasium-gitlab-service 0.2.6-1
ii ruby-gemojione 3.1.0-2
ii ruby-github-linguist 4.7.2-2
ii ruby-github-markup 1.5.1+dfsg-1
ii ruby-gitlab-flowdock-git-hook 1.0.1-2
ii ruby-gitlab-git 10.7.0-1
ii ruby-gollum-lib 4.2.1+debian-1
ii ruby-gon 6.1.0-1
ii ruby-grape 0.16.2-2
ii ruby-grape-entity 0.6.0-1
ii ruby-hamlit 2.7.5-1
ii ruby-health-check 2.4.0-1
ii ruby-hipchat 1.5.2-3
ii ruby-html-pipeline 1.11.0-1
ii ruby-htmlentities 4.3.3-1
ii ruby-httparty 0.13.7-1+deb9u1
ii ruby-influxdb 0.2.3-1
ii ruby-jquery-atwho-rails 1.3.2-2
ii ruby-jquery-rails 4.2.1-2
ii ruby-jquery-scrollto-rails 1.4.3+dfsg-1
ii ruby-jquery-turbolinks 2.1.0~dfsg-1
ii ruby-jquery-ui-rails 6.0.1+dfsg-2
ii ruby-kaminari 0.17.0-3
ii ruby-licensee 8.0.0-2
ii ruby-loofah 2.0.3-2+deb9u1
ii ruby-mail 2.6.4+dfsg1-1
ii ruby-mail-room 0.8.1-1
ii ruby-method-source 0.8.2-2
ii ruby-mousetrap-rails 1.4.6-6
ii ruby-nested-form 0.3.2-2
ii ruby-net-ssh 1:3.2.0-1
ii ruby-nokogiri 1.6.8.1-1
ii ruby-nprogress-rails 0.1.6.7-2
ii ruby-oauth2 1.2.0-1
ii ruby-octokit 4.3.0-1
ii ruby-oj 2.17.4-1
ii ruby-omniauth 1.3.1-1+deb9u1
ii ruby-omniauth-auth0 2.0.0-0+deb9u1
ii ruby-omniauth-azure-oauth2 0.0.6-1
ii ruby-omniauth-bitbucket 0.0.2-1
ii ruby-omniauth-cas3 1.1.3-1
ii ruby-omniauth-crowd 2.2.3-2
ii ruby-omniauth-facebook 4.0.0-2
ii ruby-omniauth-github 1.1.2-2
ii ruby-omniauth-gitlab 1.0.2-1
ii ruby-omniauth-google-oauth2 0.4.1-1
ii ruby-omniauth-kerberos 0.3.0-3
ii ruby-omniauth-ldap 1.0.5-1
ii ruby-omniauth-saml 1.7.0-1
ii ruby-omniauth-shibboleth 1.2.1-1
ii ruby-omniauth-twitter 1.2.1-2
ii ruby-org 0.9.12-2
ii ruby-paranoia 2.1.3-1
ii ruby-pg 0.18.4-2
ii ruby-premailer-rails 1.9.2-2
ii ruby-prof 0.16.2+dfsg-2
ii ruby-rack-attack 4.3.1-1
ii ruby-rack-cors 0.4.0-1+deb9u1
ii ruby-rack-oauth2 1.4.0-1
ii ruby-rails 2:4.2.7.1-1
ii ruby-rails-deprecated-sanitizer 1.0.3-2
ii ruby-rainbow 2.1.0-1
ii ruby-raphael-rails 2.1.2~dfsg-1
ii ruby-recaptcha 3.2.0-2
ii ruby-redcarpet 3.3.4-2
ii ruby-redcloth 4.3.2-2
ii ruby-redis-namespace 1.5.2-3
ii ruby-redis-rails 4.0.0-1
ii ruby-request-store 1.3.0-1
ii ruby-responders 2.3.0-1
ii ruby-rouge 2.0.2-1
ii ruby-rqrcode-rails3 0.1.7-1
ii ruby-sanitize 2.1.0-2
ii ruby-sass-rails 5.0.6-1
ii ruby-seed-fu 2.3.5-1
ii ruby-select2-rails 3.5.9.3-2
ii ruby-sentry-raven 2.1.2-2
ii ruby-settingslogic 2.0.9-3
ii ruby-sidekiq 4.2.3+dfsg-1
ii ruby-sidekiq-cron 0.4.2-6
ii ruby-sinatra 1.4.7-5
ii ruby-six 0.2.0-3
ii ruby-slack-notifier 1.2.1-1
ii ruby-sprockets 3.7.0-1
ii ruby-sprockets-es6 0.9.2-1
ii ruby-state-machines-activerecord 0.4.0-1
ii ruby-sys-filesystem 1.1.7-2
ii ruby-task-list 1.0.6-1
ii ruby-tinder 1.10.1-2
ii ruby-truncato 0.7.8-1
ii ruby-turbolinks 2.5.3-3
ii ruby-u2f 0.2.1-2
ii ruby-uglifier 2.7.2+dfsg-2
ii ruby-underscore-rails 1.8.2+dfsg-1
ii ruby-unf 0.1.4-2
ii ruby-unicorn-worker-killer 0.4.4-1
ii ruby-version-sorter 2.1.0+dfsg-1
ii ruby-virtus 1.0.5-3
ii ruby-vmstat 2.2.0-2
ii ruby-wikicloth 0.8.1+dfsg-4
ii ucf 3.0036
ii unicorn 5.2.0-1
Versions of packages gitlab recommends:
ii certbot 0.10.2-1
gitlab suggests no packages.
-- Configuration Files:
/etc/default/gitlab changed:
RAILS_ENV="production"
. /etc/gitlab/gitlab-debian.conf
for i in $(grep -v '#' /etc/gitlab/gitlab-debian.conf | cut -d= -f 1)
do
export $i
done
app_user=${gitlab_user}
app_root=${gitlab_app_root}
pid_path="${gitlab_pid_path}"
socket_path="${gitlab_pid_path}"
web_server_pid_path="$pid_path/unicorn.pid"
sidekiq_pid_path="$pid_path/sidekiq.pid"
sidekiq_logfile="${gitlab_log_dir}/sidekiq.log"
gitlab_workhorse_pid_path="$pid_path/gitlab-workhorse.pid"
gitlab_workhorse_options="-listenUmask 0 -listenNetwork unix -listenAddr
$socket_path/gitlab-workhorse.socket -authBackend http://127.0.0.1:8080/";
gitlab_workhorse_log="${gitlab_log_dir}/gitlab-workhorse.log"
mail_room_enabled=false
mail_room_pid_path="$pid_path/mail_room.pid"
mail_room_logfile="${gitlab_log_dir}/mail_room.log"
shell_path="/bin/bash"
/etc/gitlab/application.rb changed:
require File.expand_path('../boot', __FILE__)
require 'rails/all'
Bundler.require(:default, Rails.env)
module Gitlab
class Application < Rails::Application
require_dependency Rails.root.join('lib/gitlab/redis')
# Settings in config/environments/* take precedence over those
specified here.
# Application configuration should go into files in config/initializers
# -- all .rb files in that directory are automatically loaded.
# Sidekiq uses eager loading, but directories not in the standard Rails
# directories must be added to the eager load paths:
#
https://github.com/mperham/sidekiq/wiki/FAQ#why-doesnt-sidekiq-autoload-my-rails-application-code
# Also, there is no need to add `lib` to autoload_paths since
autoloading is
# configured to check for eager loaded paths:
#
https://github.com/rails/rails/blob/v4.2.6/railties/lib/rails/engine.rb#L687
# This is a nice reference article on autoloading/eager loading:
#
http://blog.arkency.com/2014/11/dont-forget-about-eager-load-when-extending-autoload
config.eager_load_paths.push(*%W(#{config.root}/lib
#{config.root}/app/models/ci
#{config.root}/app/models/hooks
#{config.root}/app/models/members
#{config.root}/app/models/project_services
#{config.root}/app/workers/concerns))
config.generators.templates.push("#{config.root}/generator_templates")
# Only load the plugins named here, in the order given (default is
alphabetical).
# :all can be used as a placeholder for all plugins not explicitly
named.
# config.plugins = [ :exception_notification, :ssl_requirement, :all ]
# The default locale is :en and all translations from
config/locales/*.rb,yml are auto loaded.
# config.i18n.load_path += Dir[Rails.root.join('my', 'locales',
'*.{rb,yml}').to_s]
# config.i18n.default_locale = :de
config.i18n.enforce_available_locales = false
# Configure the default encoding used in templates for Ruby 1.9.
config.encoding = "utf-8"
# Configure sensitive parameters which will be filtered from the
log file.
#
# Parameters filtered:
# - Password (:password, :password_confirmation)
# - Private tokens
# - Two-factor tokens (:otp_attempt)
# - Repo/Project Import URLs (:import_url)
# - Build variables (:variables)
# - GitLab Pages SSL cert/key info (:certificate, :encrypted_key)
# - Webhook URLs (:hook)
# - GitLab-shell secret token (:secret_token)
# - Sentry DSN (:sentry_dsn)
# - Deploy keys (:key)
config.filter_parameters += %i(
authentication_token
certificate
encrypted_key
hook
import_url
incoming_email_token
key
otp_attempt
password
password_confirmation
private_token
runners_token
secret_token
sentry_dsn
variables
)
# Enable escaping HTML in JSON.
config.active_support.escape_html_entities_in_json = true
# Use SQL instead of Active Record's schema dumper when creating
the database.
# This is necessary if your schema can't be completely dumped by
the schema dumper,
# like if you have constraints or database-specific column types
# config.active_record.schema_format = :sql
# Enable the asset pipeline
config.assets.enabled = true
config.assets.paths << Gemojione.images_path
config.assets.precompile << "*.png"
config.assets.precompile << "print.css"
config.assets.precompile << "notify.css"
config.assets.precompile << "mailers/*.css"
config.assets.precompile << "graphs/graphs_bundle.js"
config.assets.precompile << "users/users_bundle.js"
config.assets.precompile << "network/network_bundle.js"
config.assets.precompile << "profile/profile_bundle.js"
config.assets.precompile <<
"protected_branches/protected_branches_bundle.js"
config.assets.precompile << "diff_notes/diff_notes_bundle.js"
config.assets.precompile << "boards/boards_bundle.js"
config.assets.precompile << "merge_conflicts/merge_conflicts_bundle.js"
config.assets.precompile << "boards/test_utils/simulate_drag.js"
config.assets.precompile << "blob_edit/blob_edit_bundle.js"
config.assets.precompile << "snippet/snippet_bundle.js"
config.assets.precompile << "lib/utils/*.js"
config.assets.precompile << "lib/*.js"
config.assets.precompile << "u2f.js"
# Version of your assets, change this if you want to expire all
your assets
config.assets.version = '1.0'
config.action_view.sanitized_allowed_protocols = %w(smb)
config.middleware.insert_before Warden::Manager, Rack::Attack
# Allow access to GitLab API from other domains
config.middleware.insert_before Warden::Manager, Rack::Cors do
allow do
origins Gitlab.config.gitlab.url
resource '/api/*',
credentials: true,
headers: :any,
methods: :any,
expose: ['Link']
end
# Cross-origin requests must not have the session cookie available
allow do
origins '*'
resource '/api/*',
credentials: false,
headers: :any,
methods: :any,
expose: ['Link']
end
end
# Use Redis caching across all environments
redis_config_hash = Gitlab::Redis.params
redis_config_hash[:namespace] = Gitlab::Redis::CACHE_NAMESPACE
redis_config_hash[:expires_in] = 2.weeks # Cache should not grow
forever
if Sidekiq.server? # threaded context
redis_config_hash[:pool_size] = Sidekiq.options[:concurrency] + 5
redis_config_hash[:pool_timeout] = 1
end
config.cache_store = :redis_store, redis_config_hash
config.active_record.raise_in_transactional_callbacks = true
config.active_job.queue_adapter = :sidekiq
# This is needed for gitlab-shell
ENV['GITLAB_PATH_OUTSIDE_HOOK'] = ENV['PATH']
config.generators do |g|
g.factory_girl false
end
end
end
/etc/gitlab/unicorn.rb changed:
worker_processes 3
working_directory ENV['gitlab_app_root'] # available in 0.94.0+
listen "#{ENV['gitlab_pid_path']}/gitlab.socket", :backlog => 1024
listen "127.0.0.1:8080", :tcp_nopush => true
timeout 60
pid "#{ENV['gitlab_pid_path']}/unicorn.pid"
stderr_path File.join(ENV['gitlab_log_dir'],"unicorn.stderr.log")
stdout_path File.join(ENV['gitlab_log_dir'],"unicorn.stdout.log")
preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
GC.copy_on_write_friendly = true
check_client_connection false
before_fork do |server, worker|
# the following is highly recomended for Rails + "preload_app true"
# as there's no need for the master process to hold a connection
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
# The following is only recommended for memory/DB-constrained
# installations. It is not needed if your system can house
# twice as many worker_processes as you have configured.
#
# This allows a new master process to incrementally
# phase out the old master process with SIGTTOU to avoid a
# thundering herd (especially in the "preload_app false" case)
# when doing a transparent upgrade. The last worker spawned
# will then kill off the old master process with a SIGQUIT.
old_pid = "#{server.config[:pid]}.oldbin"
if old_pid != server.pid
begin
sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
Process.kill(sig, File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
end
end
#
# Throttle the master from forking too quickly by sleeping. Due
# to the implementation of standard Unix signal handlers, this
# helps (but does not completely) prevent identical, repeated signals
# from being lost when the receiving process is busy.
# sleep 1
end
after_fork do |server, worker|
# per-process listener ports for debugging/admin/migrations
# addr = "127.0.0.1:#{9293 + worker.nr}"
# server.listen(addr, :tries => -1, :delay => 5, :tcp_nopush => true)
# the following is *required* for Rails + "preload_app true",
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
# if preload_app is true, then you may also want to check and
# restart any other shared sockets/descriptors such as Memcached,
# and Redis. TokyoCabinet file handles are safe to reuse
# between any number of forked children (assuming your kernel
# correctly implements pread()/pwrite() system calls)
end
-- debconf information:
gitlab/internal/skip-preseed: false
* gitlab/remote/host: localhost
gitlab/pgsql/changeconf: false
gitlab/remote/newhost: localhost
gitlab/pgsql/manualconf:
gitlab/install-error: abort
* gitlab/fqdn: devilsf
gitlab/dbconfig-upgrade: true
gitlab/letsencrypt_email:
gitlab/internal/reconfiguring: false
gitlab/purge_data: true
* gitlab/database-type: pgsql
gitlab/remove-error: abort
gitlab/dbconfig-reinstall: false
gitlab/purge: false
gitlab/upgrade-backup: true
gitlab/pgsql/authmethod-admin: ident
gitlab/pgsql/no-empty-passwords:
gitlab/pgsql/method: TCP/IP
* gitlab/dbconfig-install: true
gitlab/remote/port:
gitlab/db/app-user: gitlab@localhost
* gitlab/ssl: true
gitlab/pgsql/authmethod-user: password
* gitlab/user: gitlab
gitlab/missing-db-package-error: abort
gitlab/upgrade-error: abort
gitlab/dbconfig-remove: true
gitlab/passwords-do-not-match:
* gitlab/letsencrypt: false
gitlab/db/dbname: gitlab_production
gitlab/pgsql/admin-user: postgres
--- End Message ---
