A little update on our story. Vacuum full was running for the whole weekend, so we didn't yet have time to rebuild indexes, because that would require more downtime, and we're not sure how long it would take. The size of the database didn't drop that much, it's now ~370Gb.
We already see some improvements. The queue doesn't get insanely large, in fact the highest peak we saw since Monday morning was about 20, but mostly it's between 0 and 3. That's good. :) One strange benefit is that we now see "Resources" and "Resource duplication" values on the Dashboard, those were previously just question marks, as you can see on my previous screenshots. However, we still get the constraint violations, steadily 2 violations per hour. They appear as "Retried" commands on the dashboard. But if the queue reaches the size 0, then does this mean these commands get written to the database eventually? The violations seem to happen shortly after puppetdb starts garbage collection. You can see it here: http://pastebin.com/B6VR67LW We are still thinking about wiping the database, and bringing up a new one, as you suggested, but that cannot be done before the next weekend. Perhaps we should rebuild the indexes first, and see if it further improves the situation. It's only been 1 day since the restart, but so far it looks a lot better then before. Cheers, ak0ska On Monday, March 4, 2013 10:55:37 PM UTC+1, Ken Barber wrote: > > Any progress today? > > On Fri, Mar 1, 2013 at 9:00 AM, ak0ska <akos....@gmail.com <javascript:>> > wrote: > > Yes, maybe not. The next step will be to recreate it from scratch. > > > > > > On Friday, March 1, 2013 5:47:06 PM UTC+1, Ken Barber wrote: > >> > >> > >> Well, I don't think a vacuum will help you - I imagine something is > >> wrong with the schema right now or some data migration failed during > >> upgrade. Esp. if you had migration issues from your custom PuppetDB. > >> Of course I can't prove this with so little knowledge - but it > >> certainly does raise a red flag. > >> > >> The only other option is to compare the schemas with a known good one, > >> and compare the database data to try and find the fault. This however > >> is prone to error and might be time-consuming. Recreating the database > >> sounds like a far more reliable option with a lot more guarantees > >> around it. > >> > >> ken. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to puppet-users...@googlegroups.com <javascript:>. > > To post to this group, send email to > > puppet...@googlegroups.com<javascript:>. > > > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
