Commented out storeconfigs/thin_storeconfigs as we were only using them to get facts in a DB and for Dashboard. Also commented out the Inventory Service stuff. Looks like that did it. Now I just need to figure out the real fix...
On Friday, September 14, 2012 10:40:16 AM UTC-4, Jason Antman wrote: > > Hi, > > I have a puppet infrastructure running 2.6.14 under Passenger with about > 30 nodes. I'm just spinning up a new Cent 6.3, puppet 2.7.19 master. Right > now I have the master built and configured using the same configuration as > my 2.6.14 setup, with a few tweaks for the new version. Puppet on the > master built itself fine. I have puppetd stopped on the master. I tried > pointing one node at it, but I'm getting: > err: Could not retrieve catalog from remote server: execution expired > > I disabled passenger and ran through the builtin webrick, and confirmed > that only one node is talking to the master. No change. No helpful > information in the master log. I'm running the master with: > /usr/sbin/puppetmasterd --no-daemonize --debug --logdest console --verbose > --trace > > and some relevant puppet.conf portions on the master: > storeconfigs = true > thin_storeconfigs = true > reports = log,puppet_dashboard > reportfrom = root > syslogfacility = user > # Remote Inventory Service > facts_terminus = rest > inventory_server = tv-puppetma1.techtarget.com # fqdn of master > inventory_port = 8140 > > The exact behavior I'm seeing is that when I run puppetd on the node, all > works well up to seeing "debug: catalog supports formats: marshal pson raw > yaml; using pson" on the node and "debug: facts supports formats: > b64_zlib_yaml pson raw yaml; using pson" on the master. At that point, it > hangs for 2 minutes (120 seconds exactly), with no output on either the > node or master. puppetmasterd is using ~25% CPU and ~14 memory. After that > time, I see the below stack trace on the master, followed by a whole slew > of the same for the facts, and "err: Could not retrieve catalog from remote > server: execution expired" on the node. > > Any ideas? I'm going to try adding resources to the master VM, but I find > it hard to believe that this is just caused by compiling a single catalog > on the master... Any advice or insight would be greatly appreciated. > > Thanks, > Jason Antman > > Trace from master: > /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill' > /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill' > /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil' > /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline' > /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line' > /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new' > /usr/lib/ruby/1.8/net/http.rb:1051:in `request' > /usr/lib/ruby/1.8/net/http.rb:1037:in `request' > /usr/lib/ruby/1.8/net/http.rb:543:in `start' > /usr/lib/ruby/1.8/net/http.rb:1035:in `request' > /usr/lib/ruby/1.8/net/http.rb:857:in `put' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `send' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `http_request' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:76:in `http_put' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:158:in `save' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:271:in `save' > /usr/lib/ruby/site_ruby/1.8/puppet/node/facts.rb:20:in `save' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:26:in > `extract_facts_from_request' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:31:in > `find' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:109:in `do_find' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in > `service' > /usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service' > /usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `call' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread' > /usr/lib/ruby/1.8/webrick/server.rb:95:in `start' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `each' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `start' > /usr/lib/ruby/1.8/webrick/server.rb:23:in `start' > /usr/lib/ruby/1.8/webrick/server.rb:82:in `start' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in > `initialize' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in > `synchronize' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start' > /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start' > /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main' > /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in > `run_command' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run' > /usr/sbin/puppetmasterd:4 > err: /usr/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill' > /usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill' > /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil' > /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline' > /usr/lib/ruby/1.8/net/http.rb:2028:in `read_status_line' > /usr/lib/ruby/1.8/net/http.rb:2017:in `read_new' > /usr/lib/ruby/1.8/net/http.rb:1051:in `request' > /usr/lib/ruby/1.8/net/http.rb:1037:in `request' > /usr/lib/ruby/1.8/net/http.rb:543:in `start' > /usr/lib/ruby/1.8/net/http.rb:1035:in `request' > /usr/lib/ruby/1.8/net/http.rb:857:in `put' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `send' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:98:in `http_request' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:76:in `http_put' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:158:in `save' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:271:in `save' > /usr/lib/ruby/site_ruby/1.8/puppet/node/facts.rb:20:in `save' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:26:in > `extract_facts_from_request' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:31:in > `find' > /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:109:in `do_find' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in > `service' > /usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service' > /usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `call' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread' > /usr/lib/ruby/1.8/webrick/server.rb:95:in `start' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `each' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `start' > /usr/lib/ruby/1.8/webrick/server.rb:23:in `start' > /usr/lib/ruby/1.8/webrick/server.rb:82:in `start' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in > `initialize' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in > `synchronize' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start' > /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start' > /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main' > /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in > `run_command' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run' > /usr/sbin/puppetmasterd:4 > err: execution expired > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/X-qhrbFle9YJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
