Jira (PDB-4339) massive failures connecting to puppetdb

2019-05-01 Thread luis ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 luis ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Austin Blatt we upgraded all our puppet servers to these versions :  
 
 
 
 
    
 
 
 puppetdb-termini-6.3.2-1.el7.noarch  
 
 
 puppetdb-6.3.2-1.el7.noarch  
 
 
 puppet-agent-6.4.2-1.el7.x86_64  
 
 
 puppetserver-6.3.0-1.el7.noarch  
 
 
 
  
 
 
 
  After the upgrade I have not seen the puppetdb issue again. do you think is related?  I took a look at https://github.com/puppetlabs/puppetdb/compare/6.3.2...master , please let me know .  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was 

Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-30 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Change By: 
 Luis Ramirez  
 
 
Attachment: 
 realtime_puppetserver.log  
 
 
Attachment: 
 realtime_puppetdb.log  
 
 
Attachment: 
 realtime_puppetdb_acess.log  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-30 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 I was able to catch the error in real time in puppetdb.log, puppetserver.log and puppetdb-access.log  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-30 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Change By: 
 Luis Ramirez  
 
 
Attachment: 
 puppetdb.log  
 
 
Attachment: 
 evidence.log  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-30 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 More evidence provided. Update : I have moved the pgsql database to the same server where puppetdb is running and upgraded pgsql to version 11. I see no improvements . I am attaching evidence.log which contains the puppetserver.log , puppetdb-acces.log and an extract from the journalctl of the VM where the error occurred and the log from puppetdb.  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 New log file attached   
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Change By: 
 Luis Ramirez  
 
 
Attachment: 
 puppetdb.log  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Change By: 
 Luis Ramirez  
 
 
Attachment: 
 puppetdb.log  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Change By: 
 Luis Ramirez  
 
 
Comment: 
 Attached new puppetdb log {code} zgrep "2019-04-24T00:03:[0-3][0-3]" puppetdb-2019-04-24.0.log.gz >> puppetdb.log{code}  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Attached new puppetdb log   
 
 
 
 
  zgrep "2019-04-24T00:03:[0-3][0-3]" puppetdb-2019-04-24.0.log.gz >> puppetdb.log
  
 
 
 
   
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Change By: 
 Luis Ramirez  
 
 
Attachment: 
 puppetdb.log  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Change By: 
 Luis Ramirez  
 
 
Attachment: 
 puppetdb.log  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Change By: 
 Luis Ramirez  
 
 
Attachment: 
 puppetdb.log  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Austin Blatt (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Austin Blatt commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 It looks like in that instance, PuppetServer received the error response at 00:03:32.297 (Format is HH:MM:SS.milliseconds) but your PuppetDB log starts at 00:03:32.990, which is 0.693 seconds after PuppetServer has already received the response. Can you attach the logs starting at 00:03:00.000 all the way to 00:03:33.000 to be sure that we can see when PuppetDB was receiving the request?  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Attached 2 log files, file named log is the error of the puppetserver and puppetdb is an extract of the puppetdb log that matches the time when the error ocurred. let me know if more details need to be attached   
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-24 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Change By: 
 Luis Ramirez  
 
 
Attachment: 
 log  
 
 
Attachment: 
 puppetdb  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-22 Thread Austin Blatt (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Austin Blatt commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Oh thanks, that's a real error with our attempt to make our jsonb gc compatible with postgres 10. But I don't believe that's the culprit with respect to your submission errors. Would you mind attaching a full PuppetDB log?  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-22 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 hi Austin Blatt the host dev-kjeon1 is just an example, it happens from time to time with different hosts . I have only found an error on the puppetdb.log file : 2019-04-22T11:45:14.721-07:00 ERROR [p.p.c.services] Error during garbage collection2019-04-22T11:45:14.721-07:00 ERROR [p.p.c.services] Error during garbage collectionjava.sql.BatchUpdateException: Batch entry 0 with recursive live_paths(path, value) as   (select key as path, value      from (select (jsonb_each(stable||volatile)).* from factsets) as base_case      union        select path||'#~'||sub_level.key as path,               sub_level.value          from live_paths,          lateral (select                      from (select (jsonb_each(value)).                             where jsonb_typeof(value) = 'object') as sub_fields                     union (select generate_series(0, jsonb_array_length(value - 1))::text as key,                                   jsonb_array_elements(value) as value                              where jsonb_typeof(value) = 'array')) as sub_level)   select path into unlogged tmp_live_paths from live_paths was aborted: ERROR: relation "tmp_live_paths" already exists  Call getNextException to see other errors in the batch. at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:148) at org.postgresql.core.ResultHandlerDelegate.handleError(ResultHandlerDelegate.java:50) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2179) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:479) at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:835) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128) at com.zaxxer.hikari.pool.HikariProxyStatement.executeBatch(HikariProxyStatement.java) at clojure.java.jdbc$execute_batch.invokeStatic(jdbc.clj:563) at clojure.java.jdbc$execute_batch.invoke(jdbc.clj:556) at clojure.java.jdbc$db_do_commands$fn_21279.invoke(jdbc.clj:906) at clojure.java.jdbc$db_transaction_STAR.invokeStatic(jdbc.clj:814) at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:741) at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:754) at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:741) at clojure.java.jdbc$db_do_commands.invokeStatic(jdbc.clj:905) at clojure.java.jdbc$db_do_commands.invoke(jdbc.clj:890) at puppetlabs.puppetdb.jdbc$do_commands.invokeStatic(jdbc.clj:38) at puppetlabs.puppetdb.jdbc$do_commands.doInvoke(jdbc.clj:33) at clojure.lang.RestFn.invoke(RestFn.java:457) at puppetlabs.puppetdb.scf.storage$delete_unused_fact_paths.invokeStatic(storage.clj:1151) at puppetlabs.puppetdb.scf.storage$delete_unused_fact_paths.invoke(storage.clj:1140) at puppetlabs.puppetdb.scf.storage$garbage_collect_BANG_$fn_27899$fn27902.invoke(storage.clj:1468) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn21879$fn21880.invoke(jdbc.clj:483) at clojure.java.jdbc$db_transaction_STAR.invokeStatic(jdbc.clj:771) at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:741) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn_21879.invoke(jdbc.clj:482) at puppetlabs.puppetdb.jdbc$fn21857$retry_sql_STAR_21862$fn21863$fn21864.invoke(jdbc.clj:454) at puppetlabs.puppetdb.jdbc$fn21857$retry_sql_STAR_21862$fn21863.invoke(jdbc.clj:453) at puppetlabs.puppetdb.jdbc$fn21857$retry_sql_STAR_21862.invoke(jdbc.clj:444) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invokeStatic(jdbc.clj:480) at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:469) at 

Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-19 Thread Austin Blatt (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Austin Blatt commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Ok so the important piece in that PuppetServer log is  
 
 
 
 
 2019-04-18T17:31:43.543-07:00 WARN [qtp853282096-1040] [puppetserver] Puppet Error connecting to puppetdb on 8081 at route /pdb/cmd/v1?checksum=f0173d7686071f91c04be589c5b605ee7d0a40f2=8=dev-kjeon1=store_report=2019-04-19T00:31:43.460Z, error message received was 'Error executing http request'. Failing over to the next PuppetDB server_url in the 'server_urls' list
  
 
 
 
  That warning means one of a few things. Most commonly, that's the message you'll see if PuppetDB is starting up. If that's the case, in the corresponding log messages for PuppetDB you'll see  
 
 
 
 
 2019-04-18T17:31:43.543-07:00 INFO [p.p.pdb-routing] HTTP request received while in maintenance mode
  
 
 
 
  Other than that, the field of options opens up a bit. One possibility is that one of the puppetservers is misconfigured in its communication to PuppetDB, but it doesn't sound like it's configured because it's not always happening. The other option is that something specific about the dev-kjeon1 manifest is causing an error (such as an invalid character). If that's the case you should certainly see an error message in PuppetDB.      
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
   

Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-18 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 2019-04-18T17:31:43.527-07:00 ERROR [qtp853282096-1040] [c.p.h.c.i.PersistentSyncHttpClient] Error executing http request 2019-04-18T17:31:43.543-07:00 WARN [qtp853282096-1040] [puppetserver] Puppet Error connecting to puppetdb on 8081 at route /pdb/cmd/v1?checksum=f0173d7686071f91c04be589c5b605ee7d0a40f2=8=dev-kjeon1=store_report=2019-04-19T00:31:43.460Z, error message received was 'Error executing http request'. Failing over to the next PuppetDB server_url in the 'server_urls' list 2019-04-18T17:31:43.551-07:00 ERROR [qtp853282096-1040] [puppetserver] Puppet Failed to execute '/pdb/cmd/v1?checksum=f0173d7686071f91c04be589c5b605ee7d0a40f2=8=dev-kjeon1=store_report=2019-04-19T00:31:43.460Z' on at least 1 of the following 'server_urls': https://puppetdb:8081 2019-04-18T17:31:43.552-07:00 ERROR [qtp853282096-1040] [puppetserver] Puppet /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/http.rb:115:in `raise_request_error' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/http.rb:156:in `failover_action' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/http.rb:216:in `action' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/command.rb:64:in `block in submit' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb/command.rb:63:in `submit' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:62:in `block in submit_command' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:59:in `submit_command' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/puppetdb.rb:24:in `block in process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/puppetdb.rb:21:in `process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:53:in `block in processors' org/jruby/RubyArray.java:1801:in `each' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:285:in `save' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:187:in `do_save' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:264:in `override' 

Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-18 Thread Austin Blatt (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Austin Blatt commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Luis Ramirez Are you seeing a warning message along with the Puppetserver error you first attached?   There are two warnings you could see  
 
 
 
 
 "Error connecting to #{server_url.host} on #{server_url.port} at route #{route}, " \  
 
 
  "error message received was '#{response.message}'. #{SERVER_URL_FAIL_MSG}"  
 
 
 
  and  
 
 
 
 
 "Error connecting to #{server_url.host} on #{server_url.port} at route #{route}, " \  
 
 
  "error message received was '#{response.message}'. #{SERVER_URL_FAIL_MSG}"  
 
 
 
      If you don't see one of those, then enable debug logging for Puppetserver and look for a message like  
 
 
 
 
 "HTTP 404 (probably normal) when connecting to #{server_url.host} on #{server_url.port} " \  
 
 
  "at route #{route}, error message received was '#{response.message}'. #{SERVER_URL_FAIL_MSG}"  
 
 
 
   
 

  
 
 
 
 

 
  

Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-18 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Austin Blatt what log level would you recommend me? I tried INFO but I am having a hard time trying to get the error. I have 3 puppet servers and I am using a load balancer, the puppet1 has setup puppetdb and has more resources than puppet2 and puppet3 , I get these puppetdb errors in the puppetserverlog in sporadic times in the puppet[1-3] servers but I do not see anything relevant in the puppetdb in puppet1   
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-11 Thread Luis Ramirez (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Luis Ramirez commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Austin Blatt yes, you are correct, those names are anonymized .   
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-11 Thread Austin Blatt (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Austin Blatt commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Luis Ramirez did you anonymize that log message ie certname=HOSTNAME and 'server_urls': https://PUPPETDBHOSTNAME:8081? If not you're likely missing some configuration for communication between Puppet Server and PuppetDB. You can follow the docs here https://puppet.com/docs/puppetdb/6.3/connect_puppet_master.html  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-11 Thread luis (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 luis commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 Austin Blatt I have changed the loglevel for puppetdb since I was not getting any relevant error. Sadly I am not able to replicate the issue by myself, I will wait and try to catch when the errors show up  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-11 Thread Austin Blatt (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 Austin Blatt commented on  PDB-4339  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
  Re: massive failures connecting to puppetdb   
 

  
 
 
 
 

 
 That looks like Puppet Server's generic log message for a failure response from PuppetDB, can you find the error from your PuppetDB log as well?  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)  
 
 

 
   
 

  
 

  
 

   





-- 
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.


Jira (PDB-4339) massive failures connecting to puppetdb

2019-04-11 Thread luis (JIRA)
Title: Message Title


 
 
 
 

 
 
 

 
   
 luis created an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 PuppetDB /  PDB-4339  
 
 
  massive failures connecting to puppetdb   
 

  
 
 
 
 

 
Issue Type: 
  Bug  
 
 
Assignee: 
 Unassigned  
 
 
Created: 
 2019/04/11 10:50 AM  
 
 
Priority: 
  Major  
 
 
Reporter: 
 luis  
 

  
 
 
 
 

 
 I managed around 1000 nodes in our stg environment and I get daily reports like the one below from different servers. My puppet server got 20GB ram and 8 CPUs . It rans puppet server and puppetdb , Java memory for puppetserver is 12 Gb and 3 Gb for puppetdb, I have tried to tunning the settings, but I have not seen improvements  2019-04-11T02:52:52.616-07:00 ERROR [qtp974514354-53945] [puppetserver] Puppet Failed to execute '/pdb/cmd/v1?checksum=9be7293b34465f977f72a402b1def5b85a041266=9=HOSTNAME=replace_catalog=2019-04-11T09:52:11.981Z' on at least 1 of the following 'server_urls': https://PUPPETDBHOSTNAME:8081 2019 PuppetDB v6.3.0  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment