There's a load balancer in front of the puppetmasters (which have SAN certs) but a single puppetdb host. Nothing save puppetmasters and my own curiosity depends on puppetdb (2.2.2) right now. (My disaster recovery plan is to rebuild the host and let the data refill by itself.)
If I need another scale-up or more things rely on puppetdb I might add database replication for PostgreSQL and a few extra puppetdb instances behind the load balancer. Given current usage I have no idea when I'd hit that point. I have 2GB max heap set and it spikes to 1.4GB usage quite rarely. On Fri, Jun 26, 2015 at 07:32:18AM -0400, Tom Tucker wrote: > Chris, > Thanks for the detailed response. BTW did you deploy HA, load balancing, > etc on the Postgres side? > Cheers > On Thu, Jun 25, 2015 at 11:11 PM, Christopher Wood > <[1][email protected]> wrote: > > Somewhere past 700 nodes (still puppetizing) our 1-core, 2GB-RAM > puppetmasters and 2-core, 2GB-RAM puppetdb host started showing signs of > overload (ssl/connect errors in agent logs, catalog/report mismatches in > puppetdb). I augmented the VMs with "hardware" to stop the complaints > and later on went off tuning. I moved the puppetmasters up to 4-core > 8GB-RAM and the puppetdb host is now 4-core 16GB-RAM. Definitely > rattling around in them now but there's lots of room for growth. > > For scaling/tuning, among others: > > > [2]https://ask.puppetlabs.com/question/13433/how-should-i-tune-passenger-to-run-puppet/ > > > [3]https://docs.puppetlabs.com/puppetdb/latest/scaling_recommendations.html > > [4]https://docs.puppetlabs.com/guides/scaling.html > > [5]http://activemq.apache.org/scaling-queues.html > > [6]http://activemq.apache.org/javalangoutofmemory.html > > After all that I analyzed catalogs across the deployment and found that > the datacat usage in the mcollective module (now > [7]https://github.com/puppet-community/puppet-mcollective) was an > abominable percentage of the total number of resources. The firewall > type ([8]https://github.com/puppetlabs/puppetlabs-firewall) was 3% of > the total resources. Since it takes less horsepower to puppet up fewer > things I figure there will be a benefit in some judicious refactoring > here and there. (Templates instead of file_line, iptables config instead > of firewall resources, et cetera.) > > However, I figure there's a benefit in a conversion effort to cram > things into puppet first and sort them out later. I had a good time just > throwing hardware at the problem to start and then tuning after the bulk > of hosts were converted to puppet management. People at companies where > incremental hardware use is expensive may want to tune early and shrink > manifests more aggressively. > On Thu, Jun 25, 2015 at 08:16:31PM -0400, Tom Tucker wrote: > > Assuming 2,500 Linux clients running Puppet community edition 3.8. > Any > > sizing recommendation for a PuppetDB system in regards to disk size > for > > the DB, CPU, Memory, etc. > > Thank you for your time and feedback. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, > send an > > email to [1][9][email protected]. > > To view this discussion on the web visit > > > > [2][10]https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com. > > For more options, visit [3][11]https://groups.google.com/d/optout. > > > > References > > > > Visible links > > 1. mailto:[12][email protected] > > 2. > > [13]https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com?utm_medium=email&utm_source=footer > > 3. [14]https://groups.google.com/d/optout > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [15][email protected]. > To view this discussion on the web visit > > [16]https://groups.google.com/d/msgid/puppet-users/20150626031113.GA832%40iniquitous.heresiarch.ca. > For more options, visit [17]https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [18][email protected]. > To view this discussion on the web visit > > [19]https://groups.google.com/d/msgid/puppet-users/CAGymF1CXG4cfFfFDEv4vRkko558un3Duo89LpqTTaHd1VmkQRQ%40mail.gmail.com. > For more options, visit [20]https://groups.google.com/d/optout. > > References > > Visible links > 1. mailto:[email protected] > 2. > https://ask.puppetlabs.com/question/13433/how-should-i-tune-passenger-to-run-puppet/ > 3. https://docs.puppetlabs.com/puppetdb/latest/scaling_recommendations.html > 4. https://docs.puppetlabs.com/guides/scaling.html > 5. http://activemq.apache.org/scaling-queues.html > 6. http://activemq.apache.org/javalangoutofmemory.html > 7. https://github.com/puppet-community/puppet-mcollective > 8. https://github.com/puppetlabs/puppetlabs-firewall > 9. mailto:puppet-users%[email protected] > 10. > https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com > 11. https://groups.google.com/d/optout > 12. mailto:puppet-users%[email protected] > 13. > https://groups.google.com/d/msgid/puppet-users/CAGymF1CQpBBFRZ2VO5x_e5XEt2sxF6Zpe_cT%2BZUaT5NbO%2BcEYA%40mail.gmail.com?utm_medium=email&utm_source=footer > 14. https://groups.google.com/d/optout > 15. mailto:puppet-users%[email protected] > 16. > https://groups.google.com/d/msgid/puppet-users/20150626031113.GA832%40iniquitous.heresiarch.ca > 17. https://groups.google.com/d/optout > 18. mailto:[email protected] > 19. > https://groups.google.com/d/msgid/puppet-users/CAGymF1CXG4cfFfFDEv4vRkko558un3Duo89LpqTTaHd1VmkQRQ%40mail.gmail.com?utm_medium=email&utm_source=footer > 20. https://groups.google.com/d/optout -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20150626132535.GA4476%40iniquitous.heresiarch.ca. For more options, visit https://groups.google.com/d/optout.
