Re: [Puppet Users] puppetdb 4.x filling up /opt/puppetlabs/server/data/puppetdb/mq

[Ryan Anderson]

It turns out this was an ID10T error. One of my puppet masters 
inadvertently got upgraded, including the puppetserver and puppetdb-termini 
RPMs. The problem went away when I reverted the puppet RPMs back to their 
previous release. From what I could tell, the newer 
puppetserver/puppetdb-termini was sending data to the puppetdb, who did not 
appreciate the format. Classic error of using two versions of softwares not 
intended to be used with each other.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/47dcd7fe-cef5-4647-87a6-9f0312daef16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] puppetdb 4.x filling up /opt/puppetlabs/server/data/puppetdb/mq

[Wyatt Alt]

Hey Ryan,

At first glance this sounds like 
https://tickets.puppetlabs.com/browse/PDB-2390, but that was fixed in 
4.0.0 so it shouldn't be the issue. The fact that you have 2 gigs under 
discarded means you are sending a significant number of commands that 
PuppetDB is unable to accept. There are numerous reasons this can happen 
-- you'll probably find more info in your logs. Given that you have so 
many discards, it could be that the bloat in your scheduler directory is 
actually legitimate (malformed commands get retried, and while being 
retried they go to scheduler.


My suggestion would be to look at your PuppetDB logs and/or the files 
under discard (these will have errors in addition to the actual message) 
to figure out the problem. Post back with the errors if need be and we 
can trouble shoot from there.


Wyatt

On 9/21/16 5:28 AM, Ryan Anderson wrote:
I'm using puppetdb-4.0.0-1.el7 (open source) and over time the 
ActiveMQ part of it has been saving a bunch of logs 
under /opt/puppetlabs/server/data/puppetdb/mq, and there does not 
appear to be any built-in log rotation, at least in the version I am 
using. After using this puppetdb a couple months, this is the sizing I 
observe:


  * 6.5GB = /opt/puppetlabs/server/data/puppetdb/mq/localhost/scheduler
  * 2GB = /opt/puppetlabs/server/data/puppetdb/mq/discard


The only solution I found online is 
this: https://docs.puppet.com/puppetdb/4.2/trouble_kahadb_corruption.html. 
The workaround there is to rename the mq directory and restart 
puppetdb, but it is intended as a troubleshoot, and I am seeking to 
save disk space and avoid regular puppetdb restarts.


What files are safe to delete? Is this still a problem if I upgrade to 
puppetdb 4.2?

--
You received this message because you are subscribed to the Google 
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to puppet-users+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/ec2b2314-cd8f-41c4-a6d6-16936ec157c9%40googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/64e20ffd-5802-32a6-e03b-0398574fbd94%40puppet.com.
For more options, visit https://groups.google.com/d/optout.