Hi Guillaume, Can you post the output of
rpm -qa | grep puppet ? It looks as if your puppetdb-terminus version could be out of sync with that of puppetdb. Wyatt On Wed, Sep 3, 2014 at 7:17 AM, Guillaume Blairon <[email protected]> wrote: > Hi, > > We're using puppetdb 2.1.0 and puppet 3.6.2 on CentOS. PuppetDB has been > working fine for a few weeks. > > Yesterday we added a new puppet master to handle our ~400 servers. When we > added the new master in the SRV records, PuppetDB began to show an erratic > behaviour : > - the number of iops on /var has raised from 50/sec to 500+/sec > - the /var/lib/puppetdb/mq/localhost/scheduler is being filled with > db-*.log files until filesystem gets full > - the logs are filled with this type of messages (~1 per second) : > --- > 2014-09-03 15:48:51,006 ERROR [c.p.p.command] > [24506b0f-6fcd-4009-9761-bfe3652ee1bd] [replace facts] Retrying after > attempt 5, due to: java.lang.IllegalArgumentException: No method in > multimethod 'process-command!' for dispatch value: ["replace facts" 3] > java.lang.IllegalArgumentException: No method in multimethod > 'process-command!' for dispatch value: ["replace facts" 3] > at clojure.lang.MultiFn.getFn(MultiFn.java:160) ~[puppetdb.jar:na] > at clojure.lang.MultiFn.invoke(MultiFn.java:231) ~[puppetdb.jar:na] > at > com.puppetlabs.puppetdb.command$produce_message_handler$fn__10782.invoke(command.clj:630) > ~[na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731$fn__10735.invoke(command.clj:537) > ~[na:na] > at > com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown > Source) ~[na:na] > at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] > at > com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731.invoke(command.clj:536) > ~[na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716$fn__10717.invoke(command.clj:490) > ~[na:na] > at > com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown > Source) ~[na:na] > at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] > at > com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716.invoke(command.clj:489) > ~[na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_command_parser$fn__10726.invoke(command.clj:512) > [na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_meter$fn__10706.invoke(command.clj:450) > [na:na] > at > com.puppetlabs.puppetdb.command$wrap_with_thread_name$fn__10740.invoke(command.clj:552) > [na:na] > at com.puppetlabs.mq$create_message_listener$reify__9909.onMessage(mq.clj:270) > [na:na] > at > org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) > [puppetdb.jar:na] > at > org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) > [puppetdb.jar:na] > at > org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) > [puppetdb.jar:na] > at > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) > [puppetdb.jar:na] > at > org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) > [puppetdb.jar:na] > at > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) > [puppetdb.jar:na] > at > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) > [puppetdb.jar:na] > at > org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) > [puppetdb.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > --- > > Until now, the only solution we have found is to stop puppetdb, > remove /var/lib/puppetdb/mq/localhost and restart puppetdb. But it's only > buying us a few hours before the filesystem gets full again. > > I'm not actually sure that the problem has a relation with the new master, > but that's more or less when the problem started. Do you have an idea of > what's happening here ? Is there something else I could check ? > > Cheers, > -- > Guillaume > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/85fb831e-4bf4-4d3d-acd3-a9d8892839e4%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/85fb831e-4bf4-4d3d-acd3-a9d8892839e4%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJDiH3EOVaQEYNOKkMz3QDCU%3Dhz2%2BrCzPn30kGQ%2BKoTWOwAOLw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
