Issue #22814 has been reported by Jiri Horky.
----------------------------------------
Bug #22814: Puppet CA: Possible race condition when requesting multiple
certificate at once and running under passenger
https://projects.puppetlabs.com/issues/22814
* Author: Jiri Horky
* Status: Unreviewed
* Priority: Normal
* Assignee:
* Category:
* Target version:
* Affected Puppet version: 3.0.2
* Keywords: Puppet, CA, certificate, race, passenger
* Branch:
----------------------------------------
We use autosign feature for all hosts in internal network. When running
multiple (e.g. 30) puppet agents for the first time at the same time, some of
them fails to generate certificates with the following error:
Info: Creating a new SSL key for ani10.domain.com
Info: Creating a new SSL certificate request for ani10.domain.com
Info: Certificate Request fingerprint (SHA256):
F3:AB:26:30:70:19:4A:6E:E5:B1:7F:B6:E1:E7:D7:B0:8B:26:AA:30:97:34:24:C0:8B:51:4A:CB:08:CC:92:A7
Exiting; failed to retrieve certificate and waitforcert is disabled
Error: Could not request certificate: Error 400 on SERVER: Could not find
certificate request for ani12.domain.com
Please note the different name in the request and in the response. Also note
that this is not an DNS issue although it is worth noting that the reverse
entry for IP address points to non-existing names. But since some clients
succeed and some not pretty much randomly, I would rule out the DNS.
On the server side, we see these errors which seems to point to some kind of a
race condition when generating certificates:
Oct 9 14:24:54 prg19 puppet-master[24018]: Signed certificate request for
ani12.domain.com
Oct 9 14:24:54 prg19 puppet-master[24043]: ani16.domain.com has a waiting
certificate request
Oct 9 14:24:54 prg19 puppet-master[24047]: ani26.domain.com has a waiting
certificate request
Oct 9 14:24:54 prg19 puppet-master[24051]: ani14.domain.com has a waiting
certificate request
Oct 9 14:24:54 prg19 puppet-master[24054]: ani10.domain.com has a waiting
certificate request
Oct 9 14:24:54 prg19 puppet-master[24057]: ani13.domain.com has a waiting
certificate request
Oct 9 14:24:56 prg19 puppet-master[24018]: Removing file
Puppet::SSL::CertificateRequest ani12.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani12.domain.com.pem'
Oct 9 14:24:56 prg19 puppet-master[24018]: ani27.domain.com has a waiting
certificate request
Oct 9 14:24:56 prg19 puppet-master[24014]: Signed certificate request for
ani25.domain.com
Oct 9 14:24:57 prg19 puppet-master[15012]: Signed certificate request for
ani25.domain.com
Oct 9 14:24:58 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani25.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani25.domain.com.pem'
Oct 9 14:24:58 prg19 puppet-master[24014]: Could not find certificate
request for ani12.domain.com
Oct 9 14:24:58 prg19 puppet-master[24014]: Removing mount "files":
"/space/puppet/files" does not exist or is not a directory
Oct 9 14:24:58 prg19 puppet-master[24018]: Could not rename
/var/lib/puppet/ssl/ca/serial to /var/lib/puppet/ssl/ca/serial.tmp: No such
file or directory - /var/lib/puppet/ssl/ca/serial.tmp or
/var/lib/puppet/ssl/ca/serial
Oct 9 14:24:58 prg19 puppet-master[15012]: Could not find certificate
request for ani12.domain.com
Oct 9 14:24:58 prg19 puppet-master[24054]: Signed certificate request for
ani25.domain.com
Oct 9 14:25:00 prg19 puppet-master[24014]: ani19.domain.com has a waiting
certificate request
Oct 9 14:25:00 prg19 puppet-master[24018]: Signed certificate request for
ani25.domain.com
Oct 9 14:25:00 prg19 puppet-master[24005]: Config file
/etc/puppet/hiera.yaml not found, using Hiera defaults
Oct 9 14:25:02 prg19 puppet-master[24027]: .tmp file already exists for
/var/lib/puppet/ssl/ca/serial; Aborting locked write. Check the .tmp file and
delete if appropriate
Oct 9 14:25:02 prg19 puppet-master[24051]: .tmp file already exists for
/var/lib/puppet/ssl/ca/serial; Aborting locked write. Check the .tmp file and
delete if appropriate
Oct 9 14:25:02 prg19 puppet-master[24047]: .tmp file already exists for
/var/lib/puppet/ssl/ca/serial; Aborting locked write. Check the .tmp file and
delete if appropriate
Oct 9 14:25:02 prg19 puppet-master[24057]: .tmp file already exists for
/var/lib/puppet/ssl/ca/serial; Aborting locked write. Check the .tmp file and
delete if appropriate
Oct 9 14:25:02 prg19 puppet-master[24043]: .tmp file already exists for
/var/lib/puppet/ssl/ca/serial; Aborting locked write. Check the .tmp file and
delete if appropriate
Oct 9 14:25:02 prg19 puppet-master[24027]: ani09.domain.com has a waiting
certificate request
Oct 9 14:25:02 prg19 puppet-master[24057]: ani08.domain.com has a waiting
certificate request
Oct 9 14:25:02 prg19 puppet-master[24008]: Signed certificate request for
ani25.domain.com
Oct 9 14:25:02 prg19 puppet-master[24014]: Signed certificate request for
ani16.domain.com
Oct 9 14:25:04 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani16.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani16.domain.com.pem'
Oct 9 14:25:04 prg19 puppet-master[24047]: ani01.domain.com has a waiting
certificate request
Oct 9 14:25:05 prg19 puppet-master[24054]: Could not rename
/var/lib/puppet/ssl/ca/serial to /var/lib/puppet/ssl/ca/serial.tmp: No such
file or directory - /var/lib/puppet/ssl/ca/serial.tmp or
/var/lib/puppet/ssl/ca/serial
Oct 9 14:25:05 prg19 puppet-master[24008]: Could not find certificate
request for ani12.domain.com
Oct 9 14:25:05 prg19 puppet-master[24057]: Signed certificate request for
ani08.domain.com
Oct 9 14:25:05 prg19 puppet-master[24008]: Removing mount "files":
"/space/puppet/files" does not exist or is not a directory
Oct 9 14:25:05 prg19 puppet-master[24054]: Signed certificate request for
ani16.domain.com
Oct 9 14:25:05 prg19 puppet-master[24008]: ani15.domain.com has a waiting
certificate request
Oct 9 14:25:05 prg19 puppet-master[15012]: Compiled catalog for
ani22.domain.com in environment production in 0.18 seconds
Oct 9 14:25:05 prg19 puppet-master[24027]: Signed certificate request for
ani16.domain.com
Oct 9 14:25:07 prg19 puppet-master[24008]: .tmp file already exists for
/var/lib/puppet/ssl/ca/serial; Aborting locked write. Check the .tmp file and
delete if appropriate
Oct 9 14:25:07 prg19 puppet-master[24057]: Removing file
Puppet::SSL::CertificateRequest ani08.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani08.domain.com.pem'
Oct 9 14:25:07 prg19 puppet-master[24057]: Could not find certificate
request for ani16.domain.com
Oct 9 14:25:07 prg19 puppet-master[24057]: ani07.domain.com has a waiting
certificate request
Oct 9 14:25:07 prg19 puppet-master[24008]: ani04.domain.com has a waiting
certificate request
Oct 9 14:25:07 prg19 puppet-master[24011]: Config file
/etc/puppet/hiera.yaml not found, using Hiera defaults
Oct 9 14:25:07 prg19 puppet-master[24005]: Compiled catalog for
ani29.domain.com in environment production in 9.16 seconds
Oct 9 14:25:07 prg19 puppet-master[24043]: Config file
/etc/puppet/hiera.yaml not found, using Hiera defaults
Oct 9 14:25:08 prg19 puppet-master[24051]: Config file
/etc/puppet/hiera.yaml not found, using Hiera defaults
Oct 9 14:25:08 prg19 puppet-master[24018]: Could not rename
/var/lib/puppet/ssl/ca/serial to /var/lib/puppet/ssl/ca/serial.tmp: No such
file or directory - /var/lib/puppet/ssl/ca/serial.tmp or
/var/lib/puppet/ssl/ca/serial
Oct 9 14:25:08 prg19 puppet-master[24014]: Signed certificate request for
ani19.domain.com
Oct 9 14:25:09 prg19 puppet-master[24047]: Could not rename
/var/lib/puppet/ssl/ca/serial to /var/lib/puppet/ssl/ca/serial.tmp: No such
file or directory - /var/lib/puppet/ssl/ca/serial.tmp or
/var/lib/puppet/ssl/ca/serial
Oct 9 14:25:09 prg19 puppet-master[24057]: Signed certificate request for
ani01.domain.com
Oct 9 14:25:11 prg19 puppet-master[24027]: .tmp file already exists for
/var/lib/puppet/ssl/ca/serial; Aborting locked write. Check the .tmp file and
delete if appropriate
Oct 9 14:25:11 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani19.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani19.domain.com.pem'
Oct 9 14:25:11 prg19 puppet-master[24018]: No such file or directory -
/var/lib/puppet/ssl/ca/serial.tmp
Oct 9 14:25:11 prg19 puppet-master[24047]: Signed certificate request for
ani08.domain.com
Oct 9 14:25:13 prg19 puppet-master[24027]: ani32.domain.com has a waiting
certificate request
Oct 9 14:25:13 prg19 puppet-master[24005]: ani31.domain.com has a waiting
certificate request
Oct 9 14:25:13 prg19 puppet-master[24008]: Signed certificate request for
ani04.domain.com
Oct 9 14:25:13 prg19 puppet-master[15012]: Compiled catalog for
ani30.domain.com in environment production in 0.23 seconds
Oct 9 14:25:13 prg19 puppet-master[24057]: Removing file
Puppet::SSL::CertificateRequest ani01.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani01.domain.com.pem'
Oct 9 14:25:13 prg19 puppet-master[24057]: Could not find certificate
request for ani19.domain.com
Oct 9 14:25:13 prg19 puppet-master[24057]: ani28.domain.com has a waiting
certificate request
Oct 9 14:25:13 prg19 puppet-master[24054]: Signed certificate request for
ani10.domain.com
Oct 9 14:25:14 prg19 puppet-master[24047]: Could not find certificate
request for ani01.domain.com
Oct 9 14:25:14 prg19 puppet-master[24014]: Signed certificate request for
ani10.domain.com
Oct 9 14:25:15 prg19 puppet-master[24008]: Removing file
Puppet::SSL::CertificateRequest ani04.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani04.domain.com.pem'
Oct 9 14:25:15 prg19 puppet-master[24054]: Removing file
Puppet::SSL::CertificateRequest ani10.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani10.domain.com.pem'
Oct 9 14:25:15 prg19 puppet-master[24008]: Could not find certificate
request for ani01.domain.com
Oct 9 14:25:15 prg19 puppet-master[24054]: Could not find certificate
request for ani12.domain.com
Oct 9 14:25:15 prg19 puppet-master[24054]: Removing mount "files":
"/space/puppet/files" does not exist or is not a directory
Oct 9 14:25:15 prg19 puppet-master[24047]: ani03.domain.com has a waiting
certificate request
Oct 9 14:25:15 prg19 puppet-master[24008]: ani06.domain.com has a waiting
certificate request
Oct 9 14:25:15 prg19 puppet-master[15012]: ani05.domain.com has a waiting
certificate request
Oct 9 14:25:15 prg19 puppet-master[24043]: Compiled catalog for
ani23.domain.com in environment production in 10.59 seconds
Oct 9 14:25:16 prg19 puppet-master[24011]: Compiled catalog for
ani20.domain.com in environment production in 11.52 seconds
Oct 9 14:25:16 prg19 puppet-master[24027]: Signed certificate request for
ani04.domain.com
Oct 9 14:25:17 prg19 puppet-master[24018]: Config file
/etc/puppet/hiera.yaml not found, using Hiera defaults
Oct 9 14:25:17 prg19 puppet-master[24043]: Removing mount "files":
"/space/puppet/files" does not exist or is not a directory
Oct 9 14:25:17 prg19 puppet-master[24005]: Signed certificate request for
ani04.domain.com
Oct 9 14:25:18 prg19 puppet-master[24051]: Compiled catalog for
ani02.domain.com in environment production in 13.08 seconds
Oct 9 14:25:18 prg19 puppet-master[24057]: Signed certificate request for
ani04.domain.com
Oct 9 14:25:19 prg19 puppet-master[24047]: Could not rename
/var/lib/puppet/ssl/ca/serial to /var/lib/puppet/ssl/ca/serial.tmp: No such
file or directory - /var/lib/puppet/ssl/ca/serial.tmp or
/var/lib/puppet/ssl/ca/serial
Oct 9 14:25:19 prg19 puppet-master[24047]: Signed certificate request for
ani03.domain.com
Oct 9 14:25:20 prg19 puppet-master[24014]: Signed certificate request for
ani13.domain.com
Oct 9 14:25:21 prg19 puppet-master[24054]: Config file
/etc/puppet/hiera.yaml not found, using Hiera defaults
Oct 9 14:25:21 prg19 puppet-master[24047]: Removing file
Puppet::SSL::CertificateRequest ani03.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani03.domain.com.pem'
Oct 9 14:25:21 prg19 puppet-master[15012]: Signed certificate request for
ani03.domain.com
Oct 9 14:25:21 prg19 puppet-master[24051]: Removing mount "files":
"/space/puppet/files" does not exist or is not a directory
Oct 9 14:25:22 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani13.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani13.domain.com.pem'
Oct 9 14:25:22 prg19 puppet-master[24008]: Could not rename
/var/lib/puppet/ssl/ca/serial to /var/lib/puppet/ssl/ca/serial.tmp: No such
file or directory - /var/lib/puppet/ssl/ca/serial.tmp or
/var/lib/puppet/ssl/ca/serial
Oct 9 14:25:22 prg19 puppet-master[24027]: Signed certificate request for
ani32.domain.com
Oct 9 14:25:22 prg19 puppet-master[24008]: Signed certificate request for
ani03.domain.com
Oct 9 14:25:22 prg19 puppet-master[24005]: Signed certificate request for
ani32.domain.com
Oct 9 14:25:22 prg19 puppet-master[24011]: Compiled catalog for
ani12.domain.com in environment production in 0.23 seconds
Oct 9 14:25:23 prg19 puppet-master[24018]: Compiled catalog for
ani21.domain.com in environment production in 7.94 seconds
Oct 9 14:25:23 prg19 puppet-master[24054]: Compiled catalog for
ani24.domain.com in environment production in 5.11 seconds
Oct 9 14:25:23 prg19 puppet-master[24047]: Signed certificate request for
ani32.domain.com
Oct 9 14:25:23 prg19 puppet-master[24014]: Signed certificate request for
ani11.domain.com
Oct 9 14:25:25 prg19 puppet-master[24005]: Removing file
Puppet::SSL::CertificateRequest ani32.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani32.domain.com.pem'
Oct 9 14:25:25 prg19 puppet-master[24047]: Removing file
Puppet::SSL::CertificateRequest ani32.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani32.domain.com.pem'
Oct 9 14:25:25 prg19 puppet-master[24027]: Removing file
Puppet::SSL::CertificateRequest ani32.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani32.domain.com.pem'
Oct 9 14:25:25 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani11.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani11.domain.com.pem'
Oct 9 14:25:25 prg19 puppet-master[24005]: Could not remove
ani32.domain.com: No such file or directory -
/var/lib/puppet/ssl/ca/requests/ani32.domain.com.pem
Oct 9 14:25:25 prg19 puppet-master[24027]: Could not remove
ani32.domain.com: No such file or directory -
/var/lib/puppet/ssl/ca/requests/ani32.domain.com.pem
Oct 9 14:25:25 prg19 puppet-master[24057]: Signed certificate request for
ani32.domain.com
Oct 9 14:25:25 prg19 puppet-master[24027]: Removing mount "files":
"/space/puppet/files" does not exist or is not a directory
Oct 9 14:25:27 prg19 puppet-master[24008]: Could not find certificate
request for ani32.domain.com
Oct 9 14:25:27 prg19 puppet-master[15012]: Signed certificate request for
ani05.domain.com
Oct 9 14:25:27 prg19 puppet-master[24047]: Signed certificate request for
ani31.domain.com
Oct 9 14:25:27 prg19 puppet-master[24057]: .tmp file already exists for
/var/lib/puppet/ssl/ca/serial; Aborting locked write. Check the .tmp file and
delete if appropriate
Oct 9 14:25:27 prg19 puppet-master[15012]: Removing file
Puppet::SSL::CertificateRequest ani05.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani05.domain.com.pem'
Oct 9 14:25:27 prg19 puppet-master[24047]: Removing file
Puppet::SSL::CertificateRequest ani31.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani31.domain.com.pem'
Oct 9 14:25:27 prg19 puppet-master[15012]: Could not find certificate
request for ani32.domain.com
Oct 9 14:25:30 prg19 puppet-master[24014]: Signed certificate request for
ani27.domain.com
Oct 9 14:25:30 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani27.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani27.domain.com.pem'
Oct 9 14:25:30 prg19 puppet-master[24047]: Signed certificate request for
ani07.domain.com
Oct 9 14:25:30 prg19 puppet-master[24047]: Removing file
Puppet::SSL::CertificateRequest ani07.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani07.domain.com.pem'
Oct 9 14:25:30 prg19 puppet-master[24047]: Signed certificate request for
ani28.domain.com
Oct 9 14:25:30 prg19 puppet-master[24047]: Removing file
Puppet::SSL::CertificateRequest ani28.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani28.domain.com.pem'
Oct 9 14:25:30 prg19 puppet-master[24047]: Could not find certificate
request for ani13.domain.com
Oct 9 14:25:30 prg19 puppet-master[24014]: Signed certificate request for
ani17.domain.com
Oct 9 14:25:30 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani17.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani17.domain.com.pem'
Oct 9 14:25:31 prg19 puppet-master[24014]: Signed certificate request for
ani18.domain.com
Oct 9 14:25:31 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani18.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani18.domain.com.pem'
Oct 9 14:25:33 prg19 puppet-master[24014]: Signed certificate request for
ani14.domain.com
Oct 9 14:25:33 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani14.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani14.domain.com.pem'
Oct 9 14:25:34 prg19 puppet-master[24014]: Signed certificate request for
ani26.domain.com
Oct 9 14:25:34 prg19 puppet-master[24014]: Removing file
Puppet::SSL::CertificateRequest ani26.domain.com at
'/var/lib/puppet/ssl/ca/requests/ani26.domain.com.pem'
Regards
Jiri Horky
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/groups/opt_out.
