Hello It is still broken.
I set soft_write_failure=false I upgraded puppet on the nodes, so now the puppet master and nodes are in version 3.4.3 This is the result of the puppetdb ssl-setup : [root@el6 lofic]# puppetdb ssl-setup PEM files in /etc/puppetdb/ssl already exists, checking integrity. Setting ssl-host in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-port in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-key in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-cert in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-ca-cert in /etc/puppetdb/conf.d/jetty.ini already correct. [root@el6 lofic]# puppetdb ssl-setup -f PEM files in /etc/puppetdb/ssl already exists, checking integrity. Overwriting existing PEM files due to -f flag Copying files: /var/lib/puppet/ssl/certs/ca.pem, /var/lib/puppet/ssl/private_keys/el6.labolinux.fr.pem and /var/lib/puppet/ssl/certs/el6.labolinux.fr.pem to /etc/puppetdb/ssl Setting ssl-host in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-port in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-key in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-cert in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-ca-cert in /etc/puppetdb/conf.d/jetty.ini already correct. I restarted the puppetdb The catalogs are still absent. When I launch the master in debug +trace mode, I see : Debug: Failed to load library 'msgpack' for feature 'msgpack' Debug: file_metadata supports formats: pson b64_zlib_yaml yaml raw /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead. Warning: ActiveRecord-based storeconfigs and inventory are deprecated. See http://links.puppetlabs.com/activerecord-deprecation (at /usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:334:in `new') Debug: Using settings: adding file resource 'dblocation': 'File[/var/lib/puppet/state/clientconfigs.sqlite3]{:path=>"/var/lib/puppet/state/clientconfigs.sqlite3", :mode=>"660", :owner=>"puppet", :group=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'railslog': 'File[/var/log/puppet/rails.log]{:path=>"/var/log/puppet/rails.log", :mode=>"600", :owner=>"puppet", :group=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Finishing transaction 23034320 Info: Connecting to sqlite3 database: /var/lib/puppet/state/clientconfigs.sqlite3 Debug: Configuring PuppetDB terminuses with config file /etc/puppet/puppetdb.conf The name resolution seems fine for the master, the puppetd and the nodes [root@el6 conf.d]# host beaker.labolinux.fr beaker.labolinux.fr has address 192.168.0.10 [root@el6 conf.d]# host 192.168.0.10 10.0.168.192.in-addr.arpa domain name pointer beaker.labolinux.fr. [root@el6 conf.d]# host el6.labolinux.fr el6.labolinux.fr has address 192.168.0.16 [root@el6 conf.d]# host 192.168.0.16 16.0.168.192.in-addr.arpa domain name pointer el6.labolinux.fr. [root@el6 conf.d]# host el6d.labolinux.fr el6d.labolinux.fr has address 192.168.0.63 [root@el6 conf.d]# host 192.168.0.63 63.0.168.192.in-addr.arpa domain name pointer el6d.labolinux.fr. I still have the SSL problem : # puppet node status el6.labolinux.fr --verbose --debug --trace Debug: Configuring PuppetDB terminuses with config file /etc/puppet/puppetdb.conf Debug: Failed to load library 'selinux' for feature 'selinux' Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppet]{:path=>"/etc/puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Puppet::Type::User::ProviderPw: file pw does not exist Debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/uuidgen does not exist Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist Debug: Failed to load library 'ldap' for feature 'ldap' Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing Debug: /User[puppet]: Provider useradd does not support features libuser; not managing attribute forcelocal Debug: Puppet::Type::Group::ProviderPw: file pw does not exist Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl does not exist Debug: Failed to load library 'ldap' for feature 'ldap' Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing Debug: /Group[puppet]: Provider groupadd does not support features libuser; not managing attribute forcelocal Debug: Using settings: adding file resource 'vardir': 'File[/var/lib/puppet]{:path=>"/var/lib/puppet", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'logdir': 'File[/var/log/puppet]{:path=>"/var/log/puppet", :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'statedir': 'File[/var/lib/puppet/state]{:path=>"/var/lib/puppet/state", :mode=>"1755", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'rundir': 'File[/var/run/puppet]{:path=>"/var/run/puppet", :mode=>"755", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'libdir': 'File[/var/lib/puppet/lib]{:path=>"/var/lib/puppet/lib", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'hiera_config': 'File[/etc/puppet/hiera.yaml]{:path=>"/etc/puppet/hiera.yaml", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'certdir': 'File[/var/lib/puppet/ssl/certs]{:path=>"/var/lib/puppet/ssl/certs", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'ssldir': 'File[/var/lib/puppet/ssl]{:path=>"/var/lib/puppet/ssl", :mode=>"771", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:path=>"/var/lib/puppet/ssl/public_keys", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'requestdir': 'File[/var/lib/puppet/ssl/certificate_requests]{:path=>"/var/lib/puppet/ssl/certificate_requests", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'privatekeydir': 'File[/var/lib/puppet/ssl/private_keys]{:path=>"/var/lib/puppet/ssl/private_keys", :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'privatedir': 'File[/var/lib/puppet/ssl/private]{:path=>"/var/lib/puppet/ssl/private", :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'hostprivkey': 'File[/var/lib/puppet/ssl/private_keys/beaker.labolinux.fr.pem]{:path=>"/var/lib/puppet/ssl/private_keys/beaker.labolinux.fr.pem", :mode=>"600", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'hostpubkey': 'File[/var/lib/puppet/ssl/public_keys/beaker.labolinux.fr.pem]{:path=>"/var/lib/puppet/ssl/public_keys/beaker.labolinux.fr.pem", :mode=>"644", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'localcacert': 'File[/var/lib/puppet/ssl/certs/ca.pem]{:path=>"/var/lib/puppet/ssl/certs/ca.pem", :mode=>"644", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'hostcrl': 'File[/var/lib/puppet/ssl/crl.pem]{:path=>"/var/lib/puppet/ssl/crl.pem", :mode=>"644", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'pluginfactdest': 'File[/var/lib/puppet/facts.d]{:path=>"/var/lib/puppet/facts.d", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet] Debug: /File[/etc/puppet/hiera.yaml]: Autorequiring File[/etc/puppet] Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/private_keys/beaker.labolinux.fr.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys] Debug: /File[/var/lib/puppet/ssl/public_keys/beaker.labolinux.fr.pem]: Autorequiring File[/var/lib/puppet/ssl/public_keys] Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] Debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/facts.d]: Autorequiring File[/var/lib/puppet] Debug: Finishing transaction 19965520 Error: Could not retrieve status for el6.labolinux.fr: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A In the puppetdb.log I see : 2014-02-28 14:13:11,984 INFO [clojure-agent-send-off-pool-2] [server.AbstractConnector] Started SelectChannelConnector@localhost:8080 2014-02-28 14:13:12,229 INFO [clojure-agent-send-off-pool-2] [ssl.SslContextFactory] Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] 2014-02-28 14:13:12,241 INFO [clojure-agent-send-off-pool-2] [server.AbstractConnector] Started [email protected]:8081 And something like : 2014-02-28 14:13:14,120 WARN [qtp1396798521-46] [io.nio] javax.net.ssl.SSLHandshakeException: null cert chain when I try the command = puppet node status When I run the agent on a node I see for example : 2014-02-28 14:19:09,268 INFO [command-proc-52] [puppetdb.command] [87024599-94b4-4b2c-a324-b8ea39d26bf0] [replace facts] el6d.labolinux.fr 2014-02-28 14:19:19,985 INFO [command-proc-52] [puppetdb.command] [bd719c3f-aeb4-41a1-98ac-a3524acb1107] [store report] puppet v3.4.3 - el6d.labolinux.fr and no errors, but still no catalogs in the db. It seems that /var/lib/puppet/state/clientconfigs.sqlite3 is refreshed after each agent run. Is this a normal store ? I had to install the activerecord and sqlite3 gems, otherwise the run of puppet on the nodes was complaining with a lack of activerecord. When I do a puppet run I see in the logs of the master : Info: Caching node for el6f.labolinux.fr Debug: Saved catalog to database in 1.19 seconds Debug: Failed to load library 'msgpack' for feature 'msgpack' Debug: catalog supports formats: pson b64_zlib_yaml yaml dot raw (... many messages like the previous 2 lines...) Debug: Failed to load library 'msgpack' for feature 'msgpack' Debug: file_metadata supports formats: pson b64_zlib_yaml yaml raw Debug: Received report to process from el6f.labolinux.fr Debug: Processing report from el6f.labolinux.fr with processor Puppet::Reports::Store Debug: Processing report from el6f.labolinux.fr with processor Puppet::Reports::Puppetdb Info: 'store report' command for el6f.labolinux.fr submitted to PuppetDB with UUID 860538fe-4f58-4b35-888b-86b1678df601 I'm stuck there. Louis 2014-02-25 14:51 GMT+01:00 Ken Barber <[email protected]>: > > with my puppetdb > > - I can't query any resource or catalog > > - exported resources are not working > > > > > > This is working : > > > > # echo '["=", ["fact", "rubyversion"], "1.8.7"]' > queryfile > > # curl -X GET http://localhost:8080/v3/nodes --data-urlencode > > query@queryfile 2>/dev/null | tail -7 > > }, { > > "name" : "el6.labolinux.fr", > > "deactivated" : null, > > "catalog_timestamp" : null, > > "facts_timestamp" : "2014-02-25T08:18:07.529Z", > > "report_timestamp" : "2014-02-25T08:18:10.018Z" > > > > I can also query with success the facts, metrics, reports > > (reports=store,puppetdb) endpoints. > > > > But it is not working with the resources or catalog endpoint : > > > > curl -X GET 'http://localhost:8080/v3/resources/User' > > -> [] > > > > curl -X GET 'http://localhost:8080/v3/resources/Package' > > -> [] > > > > curl -X GET 'http://localhost:8080/v3/resources/File' > > -> [] > > > > curl -X GET http://localhost:8080/v3/catalogs/el6.labolinux.fr > > -> { "error" : "Could not find catalog for el6.labolinux.fr" } > > > > curl -X GET http://localhost:8080/v3/nodes 2>/dev/null | grep name | > grep > > el6.labolinux.fr > > -> "name" : "el6.labolinux.fr", > > > > In addition, this is not working from the puppet master : > > > > # puppet node status el6.labolinux.fr > > Error: Could not retrieve status for el6.labolinux.fr: SSL_connect > SYSCALL > > returned=5 errno=0 state=SSLv3 read finished A > > This is the smoking gun. It looks like an SSL error is being thrown. > > > # puppet cert list el6.labolinux.fr > > + "el6.labolinux.fr" (SHA256) > > > 76:00:C9:B9:0C:31:61:9C:A5:D9:B4:49:D7:17:39:76:15:9D:18:2C:E0:07:41:6B:6C:3A:4D:68:E1:BF:65:0D > > > > I think that a consequence is that my exported resources don't work. > > > > Here is my configuration. > > > > On the master : > > > > # dpkg-query -W | egrep 'puppet(master|db)' > > puppetdb-terminus 1.6.2-1puppetlabs1 > > puppetmaster 3.4.3-1puppetlabs1 > > puppetmaster-common 3.4.3-1puppetlabs1 > > > > In puppet.conf on the master : > > > > [master] > > storeconfigs=true > > storeconfig_backend=puppetdb > > reports=store,puppetdb > > > > In routes.yaml on the master : > > > > --- > > master: > > facts: > > terminus: puppetdb > > cache: yaml > > > > In puppetdb.conf on the master : > > > > [main] > > server=el6.labolinux.fr > > port=8081 > > soft_write_failure=true > > Set the soft_write_failure to false, and you should be seeing far more > errors relating to SSL I bet. I think the problem stems from the > errors being masked and probably just being stored in the masters log. > This is the correct behaviour when this setting is true, so as to > allow the master to continue to run when PuppetDB is not operational. > > Switching to false will make the real error surface most probably. Can > you try changing that setting, restarting the puppet master and > display the results for us? Or find the error in the location where > you puppet master outputs its logs (daemon.log on Debian usually I > think?). > > > On the puppetdb node : > > > > [root@el6 ~]# rpm -qa | grep '^puppet' > > puppet-3.3.2-1.el6.noarch > > Why is your Puppet agent on the puppetdb node running an older > revision then the master? I presume you are running Puppet on the > PuppetDB node as well to manage that host correct? > > > puppetdb-1.6.2-1.el6.noarch > > > > Database backend configuration : > > > > [database] > > classname = org.postgresql.Driver > > subprotocol = postgresql > > subname = //127.0.0.1:5432/puppetdb > > username = puppetdb > > > > What am I missing ? > > This looks like a basic SSL setup issue but I have limited data to > work on. Try running "puppetdb ssl-setup", storing the results then > using "puppetdb ssl-setup -f" to force a manual repair of the > certificates PuppetDB uses. Make sure you restart PuppetDB before > trying again. > > ken. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAE4bNTmPhj2xpBiitN2e3q3%2BDmt43w%2BDLidA3j8yX_oS9h3cuA%40mail.gmail.com > . > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE9jN33eYAhA32i0zqdHi0a5bUimpQVb4oE2L8EtG3y6%2B5M%2B%3DA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
