Jira (PUP-3757) Puppet client can try to use SSL instead of TLS

[Tomas Doran (JIRA)]

Title: Message Title

Tomas Doran created an issue   Puppet / PUP-3757 Puppet client can try to use SSL instead of TLS Issue Type: Bug Affects Versions: PUP 3.7.3 Assignee: Unassigned Created: 2014/12/10 6:33 AM Environment: Ubuntu Precise physical machines. ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux] Priority: Major Reporter: Tomas Doran I'm not sure if this should be 'Critical' or 'Major', so I went with the lower - it's a crash, but it doesn't cause 100% loss of service. On some clients we seem to have issues like this: Debug: Using cached connection for https://puppet.local.yelpcorp.com:8140 Debug: Closing connection for https://puppet.local.yelpcorp.com:8140 Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server hello A: wrong version number /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/net/http.rb:920:in `connect' /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/net/http.rb:920:in `block in connect' /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout' /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/timeout.rb:101:in `call' /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/timeout.rb:101:in `timeout' /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/net/http.rb:920:in `connect' /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/net/http.rb:1451:in `begin_transport' /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/net/http.rb:1404:in `transport_request' /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/net/http.rb:1378:in `request' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/connection.rb:208:in `execute_request' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/connection.rb:176:in `block (2 levels) in request_with_redirects' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/connection.rb:219:in `block in with_connection' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/pool.rb:31:in `with_connection' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/connection.rb:218:in `with_connection' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/connection.rb:173:in `block in request_with_redirects' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/connection.rb:170:in `upto' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/connection.rb:170:in `request_with_redirects' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/connection.rb:87:in `post' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/indirector/rest.rb:83:in `http_request' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/indirector/rest.rb:66:in `http_post' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/indirector/rest.rb:94:in `block in find' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/indirector/rest.rb:190:in `block in do_request' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/indirector/request.rb:263:in `do_request' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/indirector/rest.rb:190:in `do_request' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/indirector/rest.rb:90:in `find' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/indirector/indirection.rb:201:in `find' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/configurer.rb:291:in `block in retrieve_new_catalog' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/util.rb:327:in `block in thinmark' /opt/puppet-omnibus/embedded/lib/ruby/2.1.0/benchmark.rb:294:in `realtime' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/util.rb:326:in `thinmark' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/configurer.rb:290:in `retrieve_new_catalog' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/configurer.rb:61:in `retrieve_catalog' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/configurer.rb:104:in `prepare_and_retrieve_catalog' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/configurer.rb:203:in `run_internal' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/configurer.rb:132:in `block in run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/context.rb:64:in `override' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet.rb:244:in `override' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/configurer.rb:131:in `run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/agent.rb:47:in `block (4 levels) in run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/agent/locker.rb:20:in `lock' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/agent.rb:47:in `block (3 levels) in run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/agent.rb:117:in `with_client' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/agent.rb:44:in `block (2 levels) in run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/agent.rb:82:in `run_in_fork' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/agent.rb:43:in `block in run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/application.rb:179:in `call' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/application.rb:179:in `controlled_run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/agent.rb:41:in `run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/application/agent.rb:356:in `onetime' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/application/agent.rb:322:in `run_command' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/application.rb:381:in `block (2 levels) in run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/application.rb:507:in `plugin_hook' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/application.rb:381:in `block in run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/util.rb:488:in `exit_on_fail' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/application.rb:381:in `run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/util/command_line.rb:146:in `run' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/util/command_line.rb:92:in `execute' /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/bin/puppet:8:in `<top (required)>' /opt/puppet-omnibus/embedded/bin/puppet:23:in `load' /opt/puppet-omnibus/embedded/bin/puppet:23:in `<main>' Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run It appears to be exactly what the error says - the client is somehow requesting SSL rather than TLS, and this is being refused by the web server / puppet (after POODLE fixes) This however only happens on some requests. Often the catalog compilation is fine, and instead the request to a random file will fail. Occasionally an entire run will take place without exhibiting this problem. We also have lots of machines which don't exhibit this problem (only seems to affect precise hosts). I've observed that forcing TLS in the client appears to completely fix the issue: --- /home/tdoran/factory.rb 2014-12-10 06:13:31.897450188 -0800 +++ /opt/puppet-omnibus/embedded/lib/ruby/gems/2.1.0/gems/puppet-3.7.3.y1/lib/puppet/network/http/factory.rb 2014-12-10 06:14:06.490132976 -0800 @@ -31,6 +31,7 @@   http = Net::HTTP.new(*args) http.use_ssl = site.use_ssl? + http.ssl_version = :TLSv1 # Use configured timeout (#1176) http.read_timeout = Puppet[:configtimeout] http.open_timeout = Puppet[:configtimeout] I'm not sure where to take this, or how to replicate outside our environment, as I'm unsure which part of our environment is causing the difference, and why it's only manifesting sometimes. Please feel free to reach out to me if I can help diagnose this for a better fix! Add Comment   This message was sent by Atlassian JIRA (v6.3.7#6337-sha1:2ed701e) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.