Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title luis ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb Austin Blatt we upgraded all our puppet servers to these versions : puppetdb-termini-6.3.2-1.el7.noarch puppetdb-6.3.2-1.el7.noarch puppet-agent-6.4.2-1.el7.x86_64 puppetserver-6.3.0-1.el7.noarch After the upgrade I have not seen the puppetdb issue again. do you think is related? I took a look at https://github.com/puppetlabs/puppetdb/compare/6.3.2...master , please let me know . Add Comment This message was
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez updated an issue PuppetDB / PDB-4339 massive failures connecting to puppetdb Change By: Luis Ramirez Attachment: realtime_puppetserver.log Attachment: realtime_puppetdb.log Attachment: realtime_puppetdb_acess.log Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb I was able to catch the error in real time in puppetdb.log, puppetserver.log and puppetdb-access.log Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez updated an issue PuppetDB / PDB-4339 massive failures connecting to puppetdb Change By: Luis Ramirez Attachment: puppetdb.log Attachment: evidence.log Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb More evidence provided. Update : I have moved the pgsql database to the same server where puppetdb is running and upgraded pgsql to version 11. I see no improvements . I am attaching evidence.log which contains the puppetserver.log , puppetdb-acces.log and an extract from the journalctl of the VM where the error occurred and the log from puppetdb. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb New log file attached Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez updated an issue PuppetDB / PDB-4339 massive failures connecting to puppetdb Change By: Luis Ramirez Attachment: puppetdb.log Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez updated an issue PuppetDB / PDB-4339 massive failures connecting to puppetdb Change By: Luis Ramirez Attachment: puppetdb.log Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title
Luis Ramirez updated an issue
PuppetDB / PDB-4339
massive failures connecting to puppetdb
Change By:
Luis Ramirez
Comment:
Attached new puppetdb log {code} zgrep "2019-04-24T00:03:[0-3][0-3]" puppetdb-2019-04-24.0.log.gz >> puppetdb.log{code}
Add Comment
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb Attached new puppetdb log zgrep "2019-04-24T00:03:[0-3][0-3]" puppetdb-2019-04-24.0.log.gz >> puppetdb.log Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez updated an issue PuppetDB / PDB-4339 massive failures connecting to puppetdb Change By: Luis Ramirez Attachment: puppetdb.log Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez updated an issue PuppetDB / PDB-4339 massive failures connecting to puppetdb Change By: Luis Ramirez Attachment: puppetdb.log Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez updated an issue PuppetDB / PDB-4339 massive failures connecting to puppetdb Change By: Luis Ramirez Attachment: puppetdb.log Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb Attached 2 log files, file named log is the error of the puppetserver and puppetdb is an extract of the puppetdb log that matches the time when the error ocurred. let me know if more details need to be attached Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez updated an issue PuppetDB / PDB-4339 massive failures connecting to puppetdb Change By: Luis Ramirez Attachment: log Attachment: puppetdb Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb hi Austin Blatt the host dev-kjeon1 is just an example, it happens from time to time with different hosts . I have only found an error on the puppetdb.log file : 2019-04-22T11:45:14.721-07:00 ERROR [p.p.c.services] Error during garbage collection2019-04-22T11:45:14.721-07:00 ERROR [p.p.c.services] Error during garbage collectionjava.sql.BatchUpdateException: Batch entry 0 with recursive live_paths(path, value) as (select key as path, value from (select (jsonb_each(stable||volatile)).* from factsets) as base_case union select path||'#~'||sub_level.key as path, sub_level.value from live_paths, lateral (select from (select (jsonb_each(value)). where jsonb_typeof(value) = 'object') as sub_fields union (select generate_series(0, jsonb_array_length(value - 1))::text as key, jsonb_array_elements(value) as value where jsonb_typeof(value) = 'array')) as sub_level) select path into unlogged tmp_live_paths from live_paths was aborted: ERROR: relation "tmp_live_paths" already exists Call getNextException to see other errors in the batch. at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:148) at org.postgresql.core.ResultHandlerDelegate.handleError(ResultHandlerDelegate.java:50) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2179) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:479) at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:835) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128) at com.zaxxer.hikari.pool.HikariProxyStatement.executeBatch(HikariProxyStatement.java) at clojure.java.jdbc$execute_batch.invokeStatic(jdbc.clj:563) at clojure.java.jdbc$execute_batch.invoke(jdbc.clj:556) at clojure.java.jdbc$db_do_commands$fn_21279.invoke(jdbc.clj:906) at clojure.java.jdbc$db_transaction_STAR.invokeStatic(jdbc.clj:814) at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:741) at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:754) at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:741) at clojure.java.jdbc$db_do_commands.invokeStatic(jdbc.clj:905) at clojure.java.jdbc$db_do_commands.invoke(jdbc.clj:890) at puppetlabs.puppetdb.jdbc$do_commands.invokeStatic(jdbc.clj:38) at puppetlabs.puppetdb.jdbc$do_commands.doInvoke(jdbc.clj:33) at clojure.lang.RestFn.invoke(RestFn.java:457) at puppetlabs.puppetdb.scf.storage$delete_unused_fact_paths.invokeStatic(storage.clj:1151) at puppetlabs.puppetdb.scf.storage$delete_unused_fact_paths.invoke(storage.clj:1140) at puppetlabs.puppetdb.scf.storage$garbage_collect_BANG_$fn_27899$fn27902.invoke(storage.clj:1468) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn21879$fn21880.invoke(jdbc.clj:483) at clojure.java.jdbc$db_transaction_STAR.invokeStatic(jdbc.clj:771) at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:741) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn_21879.invoke(jdbc.clj:482) at puppetlabs.puppetdb.jdbc$fn21857$retry_sql_STAR_21862$fn21863$fn21864.invoke(jdbc.clj:454) at puppetlabs.puppetdb.jdbc$fn21857$retry_sql_STAR_21862$fn21863.invoke(jdbc.clj:453) at puppetlabs.puppetdb.jdbc$fn21857$retry_sql_STAR_21862.invoke(jdbc.clj:444) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invokeStatic(jdbc.clj:480) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:469) at
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb 2019-04-18T17:31:43.527-07:00 ERROR [qtp853282096-1040] [c.p.h.c.i.PersistentSyncHttpClient] Error executing http request 2019-04-18T17:31:43.543-07:00 WARN [qtp853282096-1040] [puppetserver] Puppet Error connecting to puppetdb on 8081 at route /pdb/cmd/v1?checksum=f0173d7686071f91c04be589c5b605ee7d0a40f2=8=dev-kjeon1=store_report=2019-04-19T00:31:43.460Z, error message received was 'Error executing http request'. Failing over to the next PuppetDB server_url in the 'server_urls' list 2019-04-18T17:31:43.551-07:00 ERROR [qtp853282096-1040] [puppetserver] Puppet Failed to execute '/pdb/cmd/v1?checksum=f0173d7686071f91c04be589c5b605ee7d0a40f2=8=dev-kjeon1=store_report=2019-04-19T00:31:43.460Z' on at least 1 of the following 'server_urls': https://puppetdb:8081 2019-04-18T17:31:43.552-07:00 ERROR [qtp853282096-1040] [puppetserver] Puppet /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/http.rb:115:in `raise_request_error' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/http.rb:156:in `failover_action' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/http.rb:216:in `action' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/command.rb:64:in `block in submit' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/command.rb:63:in `submit' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:62:in `block in submit_command' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:59:in `submit_command' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/puppetdb.rb:24:in `block in process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/puppetdb.rb:21:in `process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:53:in `block in processors' org/jruby/RubyArray.java:1801:in `each' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:285:in `save' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:187:in `do_save' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:264:in `override'
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb Austin Blatt what log level would you recommend me? I tried INFO but I am having a hard time trying to get the error. I have 3 puppet servers and I am using a load balancer, the puppet1 has setup puppetdb and has more resources than puppet2 and puppet3 , I get these puppetdb errors in the puppetserverlog in sporadic times in the puppet[1-3] servers but I do not see anything relevant in the puppetdb in puppet1 Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PDB-4339) massive failures connecting to puppetdb
Title: Message Title Luis Ramirez commented on PDB-4339 Re: massive failures connecting to puppetdb Austin Blatt yes, you are correct, those names are anonymized . Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
