Jira (PDB-2170) set up the benchmarking box

[Wyatt Alt (JIRA)]

Title: Message Title

Wyatt Alt commented on PDB-2170   Re: set up the benchmarking box update on this: the puppet code is actually now deployed via r10k and lives in the puppetlabs/puppetdb-perf-box repo. Based on initial testing I've got a couple points to check with Daniel Dreier, but I think signs point to us needing to acquire some dedicated bare metal hardware. For some basic background, I have the queue on sda (~20 G) and the database on sdb (~430 G), with the Envek/pgtune module supplying # Add settings for extensions here listen_addresses = localhost effective_cache_size = 48425MB work_mem = 82646kB maintenance_work_mem = 2048MB checkpoint_segments = 32 checkpoint_completion_target = '0.9' wal_buffers = 16384kB default_statistics_target = 100 on top of the postgres defaults. This box has 60G of memory, which is the main thing informing those specific settings. Observations: It looks to me like the disks on this machine are spinning root@puppetdb-scaletest1-dev:~/work/puppetdb# lsblk -d -o name,rota sda 1 sdb 1 sr0 1 I'm not sure what the setup is behind the scenes is, so I don't know 100% that that output is accurate, but for a PDB perf testing setup I'd say SSDs (preferably separate SSDs for the queue and database) will be mandatory. Based on atop output (attached) during a benchmark run the machine is completely io bound on sda, which is the disk on which the queue currently resides. I haven't tried switching the disk because I'd rather it not contend with postgres, but if this in fact a spinning disk, and we'd like to avoid additional expense, switching to two SSDs would be a good first attempt here. The important takeaway from this with respect to PDB is that you can allocate as much memory to postgres as you want, but it does you no good if your queue disk is slow. benchmark data loading speed and consistency is way short of what it needs to be This goes hand and hand with the first bullet, but it's worth putting some comparisons down. When I run the benchmark tool on this box against a running PDB instance, I see https://gist.github.com/16434ad898b5e33ef37c In contrast, the box I typically use for benchmarking PDB is my PC. It has 24G RAM, Intel 4790 CPU, and postgres and AMQ are on separate SSDs. Postgres is configured via pgtune, but I have less RAM so the settings are more conservative than on the perf box: #------------------------------------------------------------------------------ # pgtune run on 2015-05-05 # Based on 24576664 KB RAM, platform Linux #------------------------------------------------------------------------------   maintenance_work_mem = 1408MB effective_cache_size = 16GB work_mem = 120MB wal_buffers = 16MB shared_buffers = 5632MB Based on user input I've taken to thinking that this is comparable to a hardcore production PDB setup (though others typically have a lot more RAM), but I'd obviously like to stop using it for work-related performance testing. The analogous output I get from benchmark is https://gist.github.com/wkalt/1fe91ff2f7be2e0fd4fe (this is concurrent with normal web browsing/music/etc). For completeness, here's what my work machine does on the same test: https://gist.github.com/7e38985709f3f5b25c7a I personally consider my PC to be a loose minimum bar for our internal perf testing box – building a comparable box would be well under $1000, and I think we're well within our rights to suggest a hardware upgrade to PE users with multiple tens of thousands of nodes who encounter performance issues running on less. If similar performance is possible on a VM and existing hardware, that's totally acceptable, and if anyone has optimization suggestions for the current setup we should pursue them, but as I said before I suspect we're going to end up on bare metal and IMO may as well cut to the chase. Add Comment   This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.