Hi, We're using puppetdb 2.1.0 and puppet 3.6.2 on CentOS. PuppetDB has been working fine for a few weeks.
Yesterday we added a new puppet master to handle our ~400 servers. When we added the new master in the SRV records, PuppetDB began to show an erratic behaviour : - the number of iops on /var has raised from 50/sec to 500+/sec - the /var/lib/puppetdb/mq/localhost/scheduler is being filled with db-*.log files until filesystem gets full - the logs are filled with this type of messages (~1 per second) : --- 2014-09-03 15:48:51,006 ERROR [c.p.p.command] [24506b0f-6fcd-4009-9761-bfe3652ee1bd] [replace facts] Retrying after attempt 5, due to: java.lang.IllegalArgumentException: No method in multimethod 'process-command!' for dispatch value: ["replace facts" 3] java.lang.IllegalArgumentException: No method in multimethod 'process-command!' for dispatch value: ["replace facts" 3] at clojure.lang.MultiFn.getFn(MultiFn.java:160) ~[puppetdb.jar:na] at clojure.lang.MultiFn.invoke(MultiFn.java:231) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$produce_message_handler$fn__10782.invoke(command.clj:630) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731$fn__10735.invoke(command.clj:537) ~[na:na] at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731.invoke(command.clj:536) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716$fn__10717.invoke(command.clj:490) ~[na:na] at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716.invoke(command.clj:489) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_command_parser$fn__10726.invoke(command.clj:512) [na:na] at com.puppetlabs.puppetdb.command$wrap_with_meter$fn__10706.invoke(command.clj:450) [na:na] at com.puppetlabs.puppetdb.command$wrap_with_thread_name$fn__10740.invoke(command.clj:552) [na:na] at com.puppetlabs.mq$create_message_listener$reify__9909.onMessage(mq.clj:270) [na:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) [puppetdb.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] --- Until now, the only solution we have found is to stop puppetdb, remove /var/lib/puppetdb/mq/localhost and restart puppetdb. But it's only buying us a few hours before the filesystem gets full again. I'm not actually sure that the problem has a relation with the new master, but that's more or less when the problem started. Do you have an idea of what's happening here ? Is there something else I could check ? Cheers, -- Guillaume -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/85fb831e-4bf4-4d3d-acd3-a9d8892839e4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
