Alexandros Kosiaris has submitted this change and it was merged.

Change subject: puppetmaster: servermon puppet handler concurrency improvements
......................................................................


puppetmaster: servermon puppet handler concurrency improvements

Make the transaction way smaller in order to minimize deadlocks. Also
change the natures of fact_names population to avoid the innodb
deadlocks

Change-Id: I051375d873cfe0f16c537cbac796ed92a9a37d9b
---
M modules/puppetmaster/lib/puppet/reports/servermon.rb
1 file changed, 16 insertions(+), 19 deletions(-)

Approvals:
  Alexandros Kosiaris: Verified; Looks good to me, approved



diff --git a/modules/puppetmaster/lib/puppet/reports/servermon.rb 
b/modules/puppetmaster/lib/puppet/reports/servermon.rb
index 806a574..9367bca 100644
--- a/modules/puppetmaster/lib/puppet/reports/servermon.rb
+++ b/modules/puppetmaster/lib/puppet/reports/servermon.rb
@@ -29,7 +29,6 @@
         log_level = Puppet[:log_level]
         begin
             con = Mysql.new dbserver, dbuser, dbpassword, 'puppet'
-            con.query('BEGIN')
             # First we try to update the host, if it fails, insert it
             update_host = "UPDATE hosts SET \
             environment = '#{self.environment}', \
@@ -56,33 +55,32 @@
             if File.exists?("#{Puppet[:vardir]}/yaml/facts/#{self.host}.yaml")
                 node_facts = 
YAML.load_file("#{Puppet[:vardir]}/yaml/facts/#{self.host}.yaml")
                 # We got a Ruby object, get the values attributes and walk it
+                # This part of the code causes highly concurrent queries to the
+                # DB, so extra care is taken to avoid LOCKs, deadlocks etc
                 node_facts.values.each do |key, value|
-                    # First update the fact_names table, if it fails, insert 
the
-                    # fact name
-                    update_fact_name = "UPDATE fact_names SET \
-                    updated_at = '#{self.time}' \
+                    # First try to see if the fact_name already exists
+                    con.query('BEGIN')
+                    select_fact_name = "SELECT id from fact_names \
                     WHERE name='#{key}'"
                     if log_level == 'debug'
-                        puts update_fact_name
+                        puts(select_fact_name)
                     end
-                    con.query(update_fact_name)
-                    if con.affected_rows == 0
+                    rs = con.query(select_fact_name)
+                    # So we need to insert a fact_name
+                    if rs.num_rows == 0
                         insert_fact_name = "INSERT INTO fact_names(name, 
updated_at, created_at) \
                         VALUES('#{key}', '#{self.time}', '#{self.time}')"
                         if log_level == 'debug'
-                            puts insert_fact_name
+                            puts(insert_fact_name)
                         end
                         con.query(insert_fact_name)
+                    else
+                        fact_id = rs.fetch_row[0]
+                        if log_level == 'debug'
+                            puts "Got fact: #{key} with id: #{fact_id}"
+                        end
                     end
-                    # We now can be sure the fact exists if the code reaches
-                    # this point, so just get the id
-                    query = "SELECT id from fact_names \
-                    WHERE name='#{key}'"
-                    rs = con.query(query)
-                    fact_id = rs.fetch_row[0]
-                    if log_level == 'debug'
-                        puts "Got fact: #{key} with id: #{fact_id}"
-                    end
+                    con.query('COMMIT')
                     # Now try to update the fact_value, it is fails, insert it
                     update_fact_value = "UPDATE fact_values SET \
                     updated_at = '#{self.time}', \
@@ -105,7 +103,6 @@
                     # rubocop:enable Style/Next
                 end
             end
-            con.query('COMMIT')
         rescue Mysql::Error => e
             puts "Mysql error: #{e.errno}, #{e.error}"
             puts e.errno

-- 
To view, visit https://gerrit.wikimedia.org/r/312258
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I051375d873cfe0f16c537cbac796ed92a9a37d9b
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Alexandros Kosiaris <akosia...@wikimedia.org>
Gerrit-Reviewer: Alexandros Kosiaris <akosia...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>

_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

Reply via email to