Jira (PDB-4678) PuppetDB no longer uses javax.net.ssl.trustStore to validate DB connections

[Charlie Sharpsteen (Jira)]

Title: Message Title

Charlie Sharpsteen created an issue   PuppetDB / PDB-4678 PuppetDB no longer uses javax.net.ssl.trustStore to validate DB connections Issue Type: Bug Affects Versions: PDB 6.8.0 Assignee: Unassigned Created: 2020/03/24 6:12 PM Priority: Major Reporter: Charlie Sharpsteen Prior to PuppetDB 6.8.0, PuppetDB could use the javax.net.ssl.trustStore JVM property to configure which CA certificates to use when validating database connections that use SSL. This is the first configuration method recommended by PuppetDB documentation: https://puppet.com/docs/puppetdb/latest/postgres_ssl.html#using-puppet-certificates-with-the-java-keystore This works in PuppetDB 6.7.3. After upgrading to 6.8.0, it fails silently and leaves the service in an unresponsive state. Reproduction Case Install Puppet Server 6.x on a CentOS 7 VM: yum install -y http://yum.puppetlabs.com/puppet6-release-el-7.noarch.rpm yum install -y puppetserver Bootstrap the Puppet CA and install the puppetlabs-puppetdb, puppetlabs-java_ks, and puppetlabs-concat modules: source /etc/profile.d/puppet-agent.sh puppetserver ca setup   puppet module install puppetlabs-puppetdb puppet module install puppetlabs-java_ks puppet module install puppetlabs-concat Install PuppetDB 6.7.3 and configure it to use SSL: puppet apply <<'EOF' class { 'puppetdb::globals': version => '6.7.3', }   $puppetdb_truststore = '/etc/puppetlabs/puppetdb/ssl/puppet_ca.jks'   class { 'puppetdb': database_host => $trusted['certname'], database_listen_address => '*', jdbc_ssl_properties => '?ssl=true', java_args => {'-Djavax.net.ssl.trustStore' => "=${puppetdb_truststore}", '-Djavax.net.ssl.trustStorePassword' => '=puppetdb'}, }   java_ks { 'puppetdb truststore': ensure => present, certificate => '/etc/puppetlabs/puppet/ssl/certs/ca.pem', target => $puppetdb_truststore, password => 'puppetdb', trustcacerts => true, require => Package[$puppetdb::params::puppetdb_package], before => Service[$puppetdb::params::puppetdb_service], }   file {'postgres private key': ensure => present, path => "${postgresql::params::datadir}/server.key", source => "file:///etc/puppetlabs/puppet/ssl/private_keys/${trusted['certname']}.pem", owner => 'postgres', mode => '0600', require => Package['postgresql-server'], }   concat {'postgres cert bundle': ensure => present, path => "${postgresql::params::datadir}/server.crt", owner => 'postgres', require => Package['postgresql-server'], }   concat::fragment {'agent cert': target => 'postgres cert bundle', source => "file:///etc/puppetlabs/puppet/ssl/certs/${trusted['certname']}.pem", order => '1', }   concat::fragment {'CA bundle': target => 'postgres cert bundle', source => 'file:///etc/puppetlabs/puppet/ssl/certs/ca.pem', order => '2', }   postgresql::server::config_entry {'ssl': ensure => present, value => 'on', require => [File['postgres private key'], Concat['postgres cert bundle']], } EOF Upgrade PuppetDB: yum upgrade -y puppetdb Outcome The upgrade operation hangs when package scripts re-start PuppetDB. The PuppetDB log file has a very vague message about DB connections not being available: 2020-03-25T00:58:54.288Z INFO [o.e.j.s.Server] Started @8184ms 2020-03-25T00:58:54.290Z INFO [p.p.dashboard] Redirecting / to the PuppetDB dashboard 2020-03-25T00:58:54.291Z INFO [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@713002f7{/,null,AVAILABLE} 2020-03-25T00:58:54.294Z INFO [p.t.s.s.status-core] Starting background monitoring of cpu usage metrics 2020-03-25T00:58:54.305Z INFO [p.t.s.s.status-service] Registering status callback function for service 'status-service', version 1.1.0 2020-03-25T00:58:54.305Z INFO [p.t.s.s.status-service] Registering status service HTTP API at /status 2020-03-25T00:58:54.318Z INFO [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@6abb359b{/status,null,AVAILABLE} 2020-03-25T00:58:54.329Z INFO [c.z.h.HikariDataSource] PDBWritePool - Starting... 2020-03-25T00:58:54.374Z INFO [c.z.h.HikariDataSource] PDBWritePool - Start completed. 2020-03-25T00:58:54.375Z INFO [c.z.h.HikariDataSource] PDBReadPool - Starting... 2020-03-25T00:58:54.378Z INFO [c.z.h.HikariDataSource] PDBReadPool - Start completed. 2020-03-25T00:58:54.379Z INFO [p.p.c.services] PuppetDB version 6.9.1 2020-03-25T00:58:54.379Z INFO [c.z.h.HikariDataSource] PDBMigrationsPool - Starting... 2020-03-25T00:58:54.381Z INFO [c.z.h.HikariDataSource] PDBMigrationsPool - Start completed. 2020-03-25T00:58:57.390Z ERROR [p.p.c.services] Will retry database connection after temporary failure: java.sql.SQLTransientConnectionException: PDBMigrationsPool - Connection is not available, request timed out after 3001ms. 2020-03-25T00:59:00.391Z ERROR [p.p.c.services] Will retry database connection after temporary failure: java.sql.SQLTransientConnectionException: PDBMigrationsPool - Connection is not available, request timed out after 3000ms. tcpdump shows PuppetDB connection to Postgres, but traffic stops after the database responds to the "SSL Request" message. The client should initiate a TLS handshake at this point. Expected Outcome The upgrade completes successfully. Any error with loading TLS configrion should result in a descriptive log message. Recommended Workaround Configure PuppetDB to load CA certificates using the sslrootcert parameter in the subname settings of database.ini and read_database.ini instead of using javax.net.ssl.trustStore as described here: https://puppet.com/docs/puppetdb/latest/postgres_ssl.html#using-your-own-self-signed-ca For the reproduction case above, this could be configured by: puppet apply <<'EOF' class { 'puppetdb': database_host => $trusted['certname'], database_listen_address => '*', jdbc_ssl_properties => '?ssl=true&sslrootcert=/etc/puppetlabs/puppet/ssl/certs/ca.pem', } EOF Add Comment   This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.351066.1585098730000.20234.1585098780063%40Atlassian.JIRA.