We had a similar issue and Wyatt helped me out there with huge os, mount, partition facts (I still owe you a beer or two) destroying my DB. I have dropped my DB several times to "clean" things up. We don't need the data, we just use it with Puppetboard to see things happening or not. I also had the KahaDB issue with ActiveMQ, which forced me to upgrade to 4.3.0 to get rid of it as I would fill up file systems in about 4 days with logs. Going to 4 was a big step forward as far as retries and PG errors. Used to have many many retries, now zero with 40 million plus runs. How is your catalog duplication rate/command queue/command processing/sec if I may ask? Mine is very low and I am working that issue now, but I have not had time to look into it.
On Thursday, July 6, 2017 at 11:39:41 AM UTC-5, Peter Krawetzky wrote: > > Well after several attempts at tuning the DB config and puppetdb config, > we had to drop the postgresql database and recreate it then allowing > puppetdb to create the required tables, indexes, etc. Now the command > queue is going between zero and four, processed tens of thousands of queue > commands (since restarting at 10:15am) and the cpu load on the server is > next to nothing. > > My guess is the database was corrupt somehow with all of the deletes from > the KahaDB directory plus trying to remove a huge amount of facts. I think > puppet needs to provide some type of DB reset process without have to > drop/create the DB. > > On Wednesday, June 28, 2017 at 12:25:57 PM UTC-4, Peter Krawetzky wrote: >> >> Last Sunday we hit a wall on our 3.0.2 puppetdb server. The cpu spiked >> and the KahaDB logs started to grow eventually almost filling a >> filesystem. I stopped the service, removed the mq directory per a >> troubleshooting guide, and restarted. After several minutes the same >> symptoms began again and I have not been able to come up with a puppetdb or >> postgresql config to fix this. >> >> We tried turning off storeconfig in the puppet.conf file on our puppet >> master servers but that doesn't appear to have resolved the problem. I >> also can't find a good explanation as to what this parameter really does or >> does not do even in the puppet server documentation. Anyone have a better >> insight into this? >> >> Also is there a way to just turn off puppetdb? >> >> I've attached a file that is a snapshot of the puppetdb dashboard. >> >> Anyone experience anything like this? >> > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/88e929a5-735d-4a02-b0df-4a56c28ab09d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
