[Puppet - Bug #11143] Puppet 2.6.12 fails to cope with corrupt yaml files in {facts, nodes} and hangs
Issue #11143 has been updated by Andrew Parker.
Target version deleted (2.7.x)
Bug #11143: Puppet 2.6.12 fails to cope with corrupt yaml files in {facts,
nodes} and hangs
https://projects.puppetlabs.com/issues/11143#change-80428
Author: Jo Rhett
Status: Accepted
Priority: High
Assignee:
Category: parser
Target version:
Affected Puppet version: 2.6.12
Keywords:
Branch:
So we were going through all the systems and finding ones which had corked up
due to bug #10418. As we enabled each one and brought them online, I found two
systems which could not get a catalog from the system. Debug and trace got to
evaluating the server certificate, and then called a timeout some 5 minutes
later. It never got a catalog.
I could delete the client certificate (in fact all of /var/lib/puppet) and
clean the cert on the master, create a new cert and sign it. The certificate
stuff worked fine, but the catalog wouldn't work. The puppet master never
reports compiling the catalog.
There is no inconsistency on this systeme. The affected system was axax009,
and 1-8 and 10 which are identical OS, software, same subnet, everything all
work fine. In fact, two of those are on the same VMware host as axax009.
pre
# puppet agent --test --debug --trace
debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist
debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dscl does
not exist
debug: Puppet::Type::User::ProviderPw: file pw does not exist
debug: Failed to load library 'ldap' for feature 'ldap'
debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
debug: Failed to load library 'rubygems' for feature 'rubygems'
debug: Puppet::Type::File::ProviderMicrosoft_windows: feature microsoft_windows
is missing
debug: /File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/state/graphs]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state/last_run_report.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/private_keys]
debug: /File[/var/lib/puppet/log/http.log]: Autorequiring
File[/var/lib/puppet/log]
debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/run/puppet/agent.pid]: Autorequiring File[/var/run/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/last_run_summary.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/ssl/certs/axax009.company.com.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/public_keys]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/state.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/classes.txt]: Autorequiring File[/var/lib/puppet]
debug: Finishing transaction 23772566926920
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring
[Puppet - Bug #11143] Puppet 2.6.12 fails to cope with corrupt yaml files in {facts, nodes} and hangs
Issue #11143 has been updated by Lewis Marshall.
I'm getting a very similar (same?) issue on puppet 2.7.9 (client and server).
Unfortunately deleting the yaml for this server hasn't worked.
See the stack trace below for more details:
`info: Loading facts in shadow_hostname
info: Loading facts in shadow_hostname
debug: catalog supports formats: b64_zlib_yaml dot marshal pson raw yaml; using
pson
/opt/puppet/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'
/opt/puppet/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
/opt/puppet/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/opt/puppet/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/opt/puppet/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
/opt/puppet/lib/ruby/1.8/net/http.rb:2017:in `read_new'
/opt/puppet/lib/ruby/1.8/net/http.rb:1051:in `request'
/opt/puppet/lib/ruby/1.8/net/http.rb:1037:in `request'
/opt/puppet/lib/ruby/1.8/net/http.rb:543:in `start'
/opt/puppet/lib/ruby/1.8/net/http.rb:1035:in `request'
/opt/puppet/lib/ruby/1.8/net/http.rb:845:in `post'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:94:in `send'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:94:in
`http_request'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:76:in `http_post'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:116:in `find'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:189:in
`find'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/configurer.rb:237:in
`retrieve_new_catalog'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/util.rb:476:in `thinmark'
/opt/puppet/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/util.rb:475:in `thinmark'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/configurer.rb:236:in
`retrieve_new_catalog'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/configurer.rb:86:in `retrieve_catalog'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/configurer.rb:112:in
`retrieve_and_apply_catalog'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/configurer.rb:151:in `run'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:21:in `lock'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/opt/puppet/lib/ruby/1.8/sync.rb:230:in `synchronize'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/agent.rb:103:in `with_client'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/agent.rb:37:in `run'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/application.rb:172:in `call'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/application.rb:172:in `controlled_run'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/agent.rb:35:in `run'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:336:in `onetime'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:310:in
`run_command'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in `run'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/application.rb:410:in `hook'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in `run'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/application.rb:401:in `exit_on_fail'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in `run'
/opt/puppet/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:69:in `execute'
/usr/local/bin/puppet:4
err: Could not retrieve catalog from remote server: execution expired
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
debug: Value of 'preferred_serialization_format' (pson) is invalid for report,
using default (b64_zlib_yaml)
debug: report supports formats: b64_zlib_yaml marshal raw yaml; using
b64_zlib_yaml
`
Bug #11143: Puppet 2.6.12 fails to cope with corrupt yaml files in {facts,
nodes} and hangs
https://projects.puppetlabs.com/issues/11143#change-65296
Author: Jo Rhett
Status: Accepted
Priority: High
Assignee:
Category: parser
Target version: 2.7.x
Affected Puppet version: 2.6.12
Keywords:
Branch:
So we were going through all the systems and finding ones which had corked up
due to bug #10418. As we enabled each one and brought them online, I found two
systems which could not get a catalog from the system. Debug and trace got to
evaluating the server certificate, and then called a timeout some 5 minutes
later. It never got a catalog.
I could delete the client certificate (in fact all of /var/lib/puppet) and
clean the cert on the master, create a new cert and sign it. The certificate
stuff worked fine, but the catalog wouldn't work. The puppet master never
reports compiling the catalog.
There is no inconsistency on this systeme. The affected system was axax009,
and 1-8 and 10 which are identical OS, software, same subnet, everything all
work fine. In fact, two of those are on the same VMware host as axax009.
pre
# puppet agent --test --debug --trace
debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist
debug:
[Puppet - Bug #11143] Puppet 2.6.12 fails to cope with corrupt yaml files in {facts, nodes} and hangs
Issue #11143 has been updated by Jo Rhett.
It seems odd to leave such a significant issue in the stable branch of your
code. Not everyone wants to run with the bleeding edge.
Bug #11143: Puppet 2.6.12 fails to cope with corrupt yaml files in {facts,
nodes} and hangs
https://projects.puppetlabs.com/issues/11143
Author: Jo Rhett
Status: Accepted
Priority: High
Assignee:
Category: parser
Target version: 2.7.x
Affected Puppet version: 2.6.12
Keywords:
Branch:
So we were going through all the systems and finding ones which had corked up
due to bug #10418. As we enabled each one and brought them online, I found two
systems which could not get a catalog from the system. Debug and trace got to
evaluating the server certificate, and then called a timeout some 5 minutes
later. It never got a catalog.
I could delete the client certificate (in fact all of /var/lib/puppet) and
clean the cert on the master, create a new cert and sign it. The certificate
stuff worked fine, but the catalog wouldn't work. The puppet master never
reports compiling the catalog.
There is no inconsistency on this systeme. The affected system was axax009,
and 1-8 and 10 which are identical OS, software, same subnet, everything all
work fine. In fact, two of those are on the same VMware host as axax009.
pre
# puppet agent --test --debug --trace
debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist
debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dscl does
not exist
debug: Puppet::Type::User::ProviderPw: file pw does not exist
debug: Failed to load library 'ldap' for feature 'ldap'
debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
debug: Failed to load library 'rubygems' for feature 'rubygems'
debug: Puppet::Type::File::ProviderMicrosoft_windows: feature microsoft_windows
is missing
debug: /File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/state/graphs]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state/last_run_report.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/private_keys]
debug: /File[/var/lib/puppet/log/http.log]: Autorequiring
File[/var/lib/puppet/log]
debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/run/puppet/agent.pid]: Autorequiring File[/var/run/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/last_run_summary.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/ssl/certs/axax009.company.com.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/public_keys]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/state.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/classes.txt]: Autorequiring File[/var/lib/puppet]
debug: Finishing transaction 23772566926920
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
[Puppet - Bug #11143] Puppet 2.6.12 fails to cope with corrupt yaml files in {facts, nodes} and hangs
Issue #11143 has been updated by James Turnbull.
Jo - 2.6.x is not the stable branch. 2.6.13 is the last release and the branch
is now legacy. It'll only get updates for security issues from here on. The
2.7.x branch is also not bleeding edge. We're up to the 2.7.8 release: it's
stable, functional and what we consider our stable release.
Bug #11143: Puppet 2.6.12 fails to cope with corrupt yaml files in {facts,
nodes} and hangs
https://projects.puppetlabs.com/issues/11143
Author: Jo Rhett
Status: Accepted
Priority: High
Assignee:
Category: parser
Target version: 2.7.x
Affected Puppet version: 2.6.12
Keywords:
Branch:
So we were going through all the systems and finding ones which had corked up
due to bug #10418. As we enabled each one and brought them online, I found two
systems which could not get a catalog from the system. Debug and trace got to
evaluating the server certificate, and then called a timeout some 5 minutes
later. It never got a catalog.
I could delete the client certificate (in fact all of /var/lib/puppet) and
clean the cert on the master, create a new cert and sign it. The certificate
stuff worked fine, but the catalog wouldn't work. The puppet master never
reports compiling the catalog.
There is no inconsistency on this systeme. The affected system was axax009,
and 1-8 and 10 which are identical OS, software, same subnet, everything all
work fine. In fact, two of those are on the same VMware host as axax009.
pre
# puppet agent --test --debug --trace
debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist
debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dscl does
not exist
debug: Puppet::Type::User::ProviderPw: file pw does not exist
debug: Failed to load library 'ldap' for feature 'ldap'
debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
debug: Failed to load library 'rubygems' for feature 'rubygems'
debug: Puppet::Type::File::ProviderMicrosoft_windows: feature microsoft_windows
is missing
debug: /File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/state/graphs]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state/last_run_report.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/private_keys]
debug: /File[/var/lib/puppet/log/http.log]: Autorequiring
File[/var/lib/puppet/log]
debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/run/puppet/agent.pid]: Autorequiring File[/var/run/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/last_run_summary.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/ssl/certs/axax009.company.com.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/public_keys]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/state.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/classes.txt]: Autorequiring File[/var/lib/puppet]
debug: Finishing transaction 23772566926920
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring
[Puppet - Bug #11143] Puppet 2.6.12 fails to cope with corrupt yaml files in {facts, nodes} and hangs
Issue #11143 has been updated by Nigel Kersten.
Just to make it clear, when someone gets onto this ticket, I have examples of
the corrupt YAML files.
A quick sanity checking with irb shows them just returning the boolean 'false'
when trying to use YAML.load_file.
Bug #11143: Puppet 2.6.12 fails to cope with corrupt yaml files in {facts,
nodes} and hangs
https://projects.puppetlabs.com/issues/11143
Author: Jo Rhett
Status: Unreviewed
Priority: Normal
Assignee:
Category:
Target version:
Affected Puppet version: 2.6.12
Keywords:
Branch:
So we were going through all the systems and finding ones which had corked up
due to bug #10418. As we enabled each one and brought them online, I found two
systems which could not get a catalog from the system. Debug and trace got to
evaluating the server certificate, and then called a timeout some 5 minutes
later. It never got a catalog.
I could delete the client certificate (in fact all of /var/lib/puppet) and
clean the cert on the master, create a new cert and sign it. The certificate
stuff worked fine, but the catalog wouldn't work. The puppet master never
reports compiling the catalog.
There is no inconsistency on this systeme. The affected system was axax009,
and 1-8 and 10 which are identical OS, software, same subnet, everything all
work fine. In fact, two of those are on the same VMware host as axax009.
pre
# puppet agent --test --debug --trace
debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist
debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dscl does
not exist
debug: Puppet::Type::User::ProviderPw: file pw does not exist
debug: Failed to load library 'ldap' for feature 'ldap'
debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
debug: Failed to load library 'rubygems' for feature 'rubygems'
debug: Puppet::Type::File::ProviderMicrosoft_windows: feature microsoft_windows
is missing
debug: /File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/state/graphs]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state/last_run_report.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/private_keys]
debug: /File[/var/lib/puppet/log/http.log]: Autorequiring
File[/var/lib/puppet/log]
debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/run/puppet/agent.pid]: Autorequiring File[/var/run/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/last_run_summary.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/ssl/certs/axax009.company.com.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/public_keys]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/state.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/classes.txt]: Autorequiring File[/var/lib/puppet]
debug: Finishing transaction 23772566926920
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
[Puppet - Bug #11143] Puppet 2.6.12 fails to cope with corrupt yaml files in {facts, nodes} and hangs
Issue #11143 has been updated by Nigel Kersten.
Subject changed from Some systems never get catalog, servers lock up
completely to Puppet 2.6.12 fails to cope with corrupt yaml files in
{facts, nodes} and hangs
Bug #11143: Puppet 2.6.12 fails to cope with corrupt yaml files in {facts,
nodes} and hangs
https://projects.puppetlabs.com/issues/11143
Author: Jo Rhett
Status: Unreviewed
Priority: Normal
Assignee:
Category:
Target version:
Affected Puppet version: 2.6.12
Keywords:
Branch:
So we were going through all the systems and finding ones which had corked up
due to bug #10418. As we enabled each one and brought them online, I found two
systems which could not get a catalog from the system. Debug and trace got to
evaluating the server certificate, and then called a timeout some 5 minutes
later. It never got a catalog.
I could delete the client certificate (in fact all of /var/lib/puppet) and
clean the cert on the master, create a new cert and sign it. The certificate
stuff worked fine, but the catalog wouldn't work. The puppet master never
reports compiling the catalog.
There is no inconsistency on this systeme. The affected system was axax009,
and 1-8 and 10 which are identical OS, software, same subnet, everything all
work fine. In fact, two of those are on the same VMware host as axax009.
pre
# puppet agent --test --debug --trace
debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist
debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dscl does
not exist
debug: Puppet::Type::User::ProviderPw: file pw does not exist
debug: Failed to load library 'ldap' for feature 'ldap'
debug: Puppet::Type::User::ProviderLdap: feature ldap is missing
debug: Failed to load library 'rubygems' for feature 'rubygems'
debug: Puppet::Type::File::ProviderMicrosoft_windows: feature microsoft_windows
is missing
debug: /File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/state/graphs]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/clientbucket]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state/last_run_report.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_data]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/client_yaml]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/private_keys]
debug: /File[/var/lib/puppet/log/http.log]: Autorequiring
File[/var/lib/puppet/log]
debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/run/puppet/agent.pid]: Autorequiring File[/var/run/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/last_run_summary.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/ssl/private]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/ssl/certs/axax009.company.com.pem]: Autorequiring
File[/var/lib/puppet/ssl/certs]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys/axax009.company.com.pem]:
Autorequiring File[/var/lib/puppet/ssl/public_keys]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/state/state.yaml]: Autorequiring
File[/var/lib/puppet/state]
debug: /File[/var/lib/puppet/classes.txt]: Autorequiring File[/var/lib/puppet]
debug: Finishing transaction 23772566926920
debug: /File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet]
debug: /File[/var/lib/puppet/log]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet]
debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring
File[/var/lib/puppet/ssl]
debug: /File[/var/lib/puppet/ssl/private
