Re: [PERFORM] More shared buffers causes lower performances
On Dec 27, 2007 7:54 PM, Tom Lane [EMAIL PROTECTED] wrote: I concur with Greg Stark's earlier comment that this is all overreaction. Let's just fix the misleading comment in the documentation and leave it at that. IMHO, we should also have a special tag for all the binaries distributed with these options on the official website (RPM or not). If the RPM packages' version has been tagged .debug or something like that, it would have been the first thing I checked. I like Gregory's idea to add a warning in pgbench. I usually run a few pgbench tests to check there is no obvious problem even if I use another more complicated benchmark afterwards. I don't know if that's what other people do, though. -- Guillaume ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PERFORM] More shared buffers causes lower performances
On Dec 25, 2007 7:06 PM, Guillaume Smet [EMAIL PROTECTED] wrote: While monitoring the server with vmstat, I can't see any real reason why it's slower. When shared_buffers has a higher value, I/O are lower, context switches too and finally performances. The CPU usage is quite similar (~50-60%). I/O doesn't limit the performances AFAICS. Can you confirm that i/o is lower according to iostat? One possibility is that you are on the cusp of where your server's memory covers the database and the higher buffers results in lower memory efficiency. If raising shared buffers is getting you more page faults to disk, this would explain the lower figures regardless of the # of syscalls. If your iowait is zero though the test is cpu bound and this distinction is moot. merlin ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PERFORM] More shared buffers causes lower performances
On Thu, Dec 27, 2007 at 01:10:29AM -0500, Tom Lane wrote: Greg Smith [EMAIL PROTECTED] writes: On Wed, 26 Dec 2007, Guillaume Smet wrote: beta RPMs are by default compiled with --enable-debug and --enable-cassert which doesn't help them to fly fast... Got that right. Last time I was going crazy after running pgbench with those options and not having realized what I changed, I was getting a 50% slowdown on results that way compared to without the debugging stuff. Didn't realize it scaled with shared_buffers though. See AtEOXact_Buffers(). There are probably any number of other interesting scaling behaviors --- in my tests, AllocSetCheck() is normally a major cycle-eater if --enable-cassert is set, and that costs time proportional to the number of memory chunks allocated by the query. Currently the docs say that --enable-cassert Enables firsttermassertion/ checks in the server, which test for many quotecannot happen/ conditions. This is invaluable for code development purposes, but the tests slow things down a little. Maybe we ought to put that more strongly --- s/a little/significantly/, perhaps? Sounds like a good idea. We got bit by the same thing when doing some benchmarks on the MSVC port (and with we I mean Dave did the work, and several people couldn't understand why the numbers sucked) //Magnus ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PERFORM] More shared buffers causes lower performances
On Dec 27, 2007 7:10 AM, Tom Lane [EMAIL PROTECTED] wrote: Enables firsttermassertion/ checks in the server, which test for many quotecannot happen/ conditions. This is invaluable for code development purposes, but the tests slow things down a little. Maybe we ought to put that more strongly --- s/a little/significantly/, perhaps? +1. It seems closer to the reality. -- Guillaume ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PERFORM] More shared buffers causes lower performances
Tom Lane escribió: Currently the docs say that --enable-cassert Enables firsttermassertion/ checks in the server, which test for many quotecannot happen/ conditions. This is invaluable for code development purposes, but the tests slow things down a little. Maybe we ought to put that more strongly --- s/a little/significantly/, perhaps? I don't think it will make any difference, because people don't read configure documentation. They read configure --help. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PERFORM] More shared buffers causes lower performances
Alvaro Herrera [EMAIL PROTECTED] writes: Tom Lane escribió: Currently the docs say that --enable-cassert Enables firsttermassertion/ checks in the server, which test for many quotecannot happen/ conditions. This is invaluable for code development purposes, but the tests slow things down a little. Maybe we ought to put that more strongly --- s/a little/significantly/, perhaps? I don't think it will make any difference, because people don't read configure documentation. They read configure --help. Fwiw I think you're all getting a bit caught up in this one context. While the slowdown is significant when you take out the stopwatch, under normal interactive use you're not going to notice your queries being especially slow. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] More shared buffers causes lower performances
On Thu, 27 Dec 2007, Gregory Stark wrote: Fwiw I think you're all getting a bit caught up in this one context. I lost a day once over this problem. Guillaume lost at least that much. Sounds like Magnus and Dave got a good sized dose as well. Seems like something worth warning people about to me. The worst time people can run into a performance regression is when they're running a popular benchmarking tool. I didn't think this was a big problem because I thought it was limited to developers who shot their own foot, but if there are packagers turning this on to improve beta feedback it deserves some wider mention. As for the suggestion that people don't read the documentation, take a look at the above list of developers and tell me whether that group is aware of what's in the docs or not. I had never seen anyone bring this up before I ran into it, and I dumped a strong warning into http://developer.postgresql.org/index.php/Working_with_CVS#Initial_setup so at least it was written down somewhere. -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PERFORM] More shared buffers causes lower performances
Greg Smith [EMAIL PROTECTED] writes: ... I didn't think this was a big problem because I thought it was limited to developers who shot their own foot, but if there are packagers turning this on to improve beta feedback it deserves some wider mention. Yeah, binary packages that are built with --enable-cassert perhaps need to be labeled as not intended for benchmarking or some such. regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PERFORM] More shared buffers causes lower performances
Greg Smith [EMAIL PROTECTED] writes: The worst time people can run into a performance regression is when they're running a popular benchmarking tool. Hm, perhaps pg_bench should do a show debug_assertions and print a warning if the answer isn't off. We could encourage other benchmark software to do something similar. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's RemoteDBA services! ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PERFORM] More shared buffers causes lower performances
Alvaro Herrera [EMAIL PROTECTED] writes: Perhaps make them emit a WARNING at server start or something. I concur with Greg Stark's earlier comment that this is all overreaction. Let's just fix the misleading comment in the documentation and leave it at that. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PERFORM] More shared buffers causes lower performances
Tom Lane escribió: Greg Smith [EMAIL PROTECTED] writes: ... I didn't think this was a big problem because I thought it was limited to developers who shot their own foot, but if there are packagers turning this on to improve beta feedback it deserves some wider mention. Yeah, binary packages that are built with --enable-cassert perhaps need to be labeled as not intended for benchmarking or some such. Perhaps make them emit a WARNING at server start or something. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PERFORM] More shared buffers causes lower performances
On Wed, 2007-12-26 at 01:06 +0100, Guillaume Smet wrote: I lowered the number of concurrent clients to 50 because 100 is quite high and I obtain the same sort of results: shared_buffers=32MB: 1869 tps shared_buffers=64MB: 1844 tps shared_buffers=512MB: 1676 tps shared_buffers=1024MB: 1559 tps Can you try with bgwriter_lru_maxpages = 0 So we can see if the bgwriter has any hand in this? -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PERFORM] More shared buffers causes lower performances
Guillaume Smet a écrit : Hi all, I'm currently benchmarking the new PostgreSQL server of one of our customers with PostgreSQL 8.3 beta4. I have more or less the same configuration Stefan tested in his blog [1]: - Dell 2900 with two brand new X5365 processors (quad core 3.0 GHz), 16 GB of memory - a RAID1 array for pg_xlog and a 6 disks RAID10 array for data (I moved pg_xlog to the RAID10 array for a few runs - same behaviour) - all 73 GB 15k drives - CentOS 5.1 - 64 bits Which kernel do you have ? I started working on pgbench tests. I made a not so stupid configuration to begin with and I was quite disappointed by my results compared to Stefan's. I decided to test with a more default shared_buffers configuration to be able to compare my results with Stefan's graph [2]. And the fact is that with a very low shared_buffers configuration, my results are quite similar to Stefan's results but, as soon as I put higher values of shared_buffers, performances begins degrading [3]. I performed my tests with: pgbench -i -s 100 -U postgres bench and pgbench -s 100 -c 100 -t 3 -U postgres bench. Of course, I initialize the database before each run. I made my tests in one direction then in the other with similar results so it's not a degradation due to consecutive runs. I lowered the number of concurrent clients to 50 because 100 is quite high and I obtain the same sort of results: shared_buffers=32MB: 1869 tps shared_buffers=64MB: 1844 tps shared_buffers=512MB: 1676 tps shared_buffers=1024MB: 1559 tps Non default parameters are: max_connections = 200 work_mem = 32MB wal_buffers = 1024kB checkpoint_segments = 192 effective_cache_size = 5GB (I use more or less the configuration used by Stefan - I had the same behaviour with default wal_buffers and checkpoint_segments) While monitoring the server with vmstat, I can't see any real reason why it's slower. When shared_buffers has a higher value, I/O are lower, context switches too and finally performances. The CPU usage is quite similar (~50-60%). I/O doesn't limit the performances AFAICS. I must admit I'm a bit puzzled. Does anyone have any pointer which could explain this behaviour or any way to track the issue? I'll be glad to perform any test needed to understand the problem. Thanks. [1] http://www.kaltenbrunner.cc/blog/index.php?/archives/21-8.3-vs.-8.2-a-simple-benchmark.html [2] http://www.kaltenbrunner.cc/blog/uploads/83b4shm.gif [3] http://people.openwide.fr/~gsmet/postgresql/tps_shared_buffers.png (X=shared_buffers in MB/Y=results with pgbench) -- Guillaume ---(end of broadcast)--- TIP 6: explain analyze is your friend -- Cédric Villemain Administrateur de Base de Données Cel: +33 (0)6 74 15 56 53 http://dalibo.com - http://dalibo.org begin:vcard fn;quoted-printable:C=C3=A9dric Villemain n;quoted-printable:Villemain;C=C3=A9dric org:Dalibo email;internet:[EMAIL PROTECTED] title:Consultant PostgreSQL tel;cell:+33 (0)6 74 15 56 53 x-mozilla-html:FALSE url:dalibo.com version:2.1 end:vcard ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PERFORM] More shared buffers causes lower performances
On Dec 26, 2007 12:21 PM, Simon Riggs [EMAIL PROTECTED] wrote: Can you try with bgwriter_lru_maxpages = 0 So we can see if the bgwriter has any hand in this? I will. I'm currently running tests with less concurrent clients (16) with exactly the same results: 64M 4213.314902 256M 4012.782820 512M 3676.840722 768M 3377.791211 1024M 2863.133965 64M again 4274.531310 I'm rerunning the tests using Greg Smith's pgbench-tools [1] to obtain a graph of each run. [1] http://www.westnet.com/~gsmith/content/postgresql/pgbench-tools.htm -- Guillaume ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PERFORM] More shared buffers causes lower performances
On Dec 26, 2007 12:06 PM, Cédric Villemain [EMAIL PROTECTED] wrote: Which kernel do you have ? Kernel of the distro. So a RH flavoured 2.6.18. -- Guillaume ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] More shared buffers causes lower performances
On Dec 26, 2007 12:21 PM, Simon Riggs [EMAIL PROTECTED] wrote: bgwriter_lru_maxpages = 0 So we can see if the bgwriter has any hand in this? It doesn't change the behaviour I have. It's not checkpointing either as using pgbench-tools, I can see that tps and latency are quite stable during the entire run. Btw, thanks Greg for these nice tools. I thought it may be some sort of lock contention so I made a few tests with -N but I have the same behaviour. Then I decided to perform read-only tests using -S option (pgbench -S -s 100 -c 16 -t 3 -U postgres bench). And still the same behaviour: shared_buffers=64MB : 20k tps shared_buffers=1024MB : 8k tps Any other idea? -- Guillaume ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PERFORM] More shared buffers causes lower performances
Hello I tested it and it is true. In my configuration 1GRam, Fedora 8, is PostgreSQL most fast with 32M shared buffers :(. Diff is about 5% to 256M Regards Pavel Stehule On 26/12/2007, Guillaume Smet [EMAIL PROTECTED] wrote: On Dec 26, 2007 12:21 PM, Simon Riggs [EMAIL PROTECTED] wrote: bgwriter_lru_maxpages = 0 So we can see if the bgwriter has any hand in this? It doesn't change the behaviour I have. It's not checkpointing either as using pgbench-tools, I can see that tps and latency are quite stable during the entire run. Btw, thanks Greg for these nice tools. I thought it may be some sort of lock contention so I made a few tests with -N but I have the same behaviour. Then I decided to perform read-only tests using -S option (pgbench -S -s 100 -c 16 -t 3 -U postgres bench). And still the same behaviour: shared_buffers=64MB : 20k tps shared_buffers=1024MB : 8k tps Any other idea? -- Guillaume ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PERFORM] More shared buffers causes lower performances
On Dec 26, 2007 4:41 PM, Guillaume Smet [EMAIL PROTECTED] wrote: Then I decided to perform read-only tests using -S option (pgbench -S -s 100 -c 16 -t 3 -U postgres bench). And still the same behaviour: shared_buffers=64MB : 20k tps shared_buffers=1024MB : 8k tps Some more information. If I strace the backends during the test, the test is faster with shared_buffers=1024MB and I have less system calls (less read and less lseek). A quick cut | uniq | sort gives me: With 64MB: 12548 semop 160039 sendto 160056 recvfrom 294289 read 613338 lseek With 1024MB: 11396 semop 129947 read 160039 sendto 160056 recvfrom 449584 lseek -- Guillaume ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PERFORM] More shared buffers causes lower performances
On Wed, 26 Dec 2007, Guillaume Smet wrote: It's not checkpointing either as using pgbench-tools, I can see that tps and latency are quite stable during the entire run. Btw, thanks Greg for these nice tools. I stole the graph idea from Mark Wong's DBT2 code and one of these days I'll credit him appropriately. Then I decided to perform read-only tests using -S option (pgbench -S -s 100 -c 16 -t 3 -U postgres bench). And still the same behaviour: shared_buffers=64MB : 20k tps shared_buffers=1024MB : 8k tps Ah, now this is really interesting, as it rules out all the write components and should be easy to replicate even on a smaller server. As you've already dumped a bunch of time into this the only other thing I would suggest checking is whether the same behavior also happens on 8.2 on your server. -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PERFORM] More shared buffers causes lower performances
On Dec 26, 2007 7:23 PM, Greg Smith [EMAIL PROTECTED] wrote: Ah, now this is really interesting, as it rules out all the write components and should be easy to replicate even on a smaller server. As you've already dumped a bunch of time into this the only other thing I would suggest checking is whether the same behavior also happens on 8.2 on your server. Let's go with 8.2.5 on the same server (-s 100 / 16 clients / 50k transactions per client / only read using -S option): 64MB: 33814 tps 512MB: 35833 tps 1024MB: 36986 tps It's more consistent with what I expected. I used PGDG RPMs compiled by Devrim for 8.2.5 and the ones I compiled myself for 8.3b4 (based on the src.rpm of Devrim). I just asked Devrim to build a set of x86_64 RPMs for 8.3b4 to see if it's not a compilation problem (they were compiled on a brand new box freshly installed so it would be a bit surprising but I want to be sure). He's kindly uploading them right now so I'll work on new tests using his RPMs. I'll keep you informed of the results. -- Guillaume ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PERFORM] More shared buffers causes lower performances
Greg Smith [EMAIL PROTECTED] writes: On Wed, 26 Dec 2007, Guillaume Smet wrote: beta RPMs are by default compiled with --enable-debug and --enable-cassert which doesn't help them to fly fast... Got that right. Last time I was going crazy after running pgbench with those options and not having realized what I changed, I was getting a 50% slowdown on results that way compared to without the debugging stuff. Didn't realize it scaled with shared_buffers though. See AtEOXact_Buffers(). There are probably any number of other interesting scaling behaviors --- in my tests, AllocSetCheck() is normally a major cycle-eater if --enable-cassert is set, and that costs time proportional to the number of memory chunks allocated by the query. Currently the docs say that --enable-cassert Enables firsttermassertion/ checks in the server, which test for many quotecannot happen/ conditions. This is invaluable for code development purposes, but the tests slow things down a little. Maybe we ought to put that more strongly --- s/a little/significantly/, perhaps? regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
[PERFORM] More shared buffers causes lower performances
Hi all, I'm currently benchmarking the new PostgreSQL server of one of our customers with PostgreSQL 8.3 beta4. I have more or less the same configuration Stefan tested in his blog [1]: - Dell 2900 with two brand new X5365 processors (quad core 3.0 GHz), 16 GB of memory - a RAID1 array for pg_xlog and a 6 disks RAID10 array for data (I moved pg_xlog to the RAID10 array for a few runs - same behaviour) - all 73 GB 15k drives - CentOS 5.1 - 64 bits I started working on pgbench tests. I made a not so stupid configuration to begin with and I was quite disappointed by my results compared to Stefan's. I decided to test with a more default shared_buffers configuration to be able to compare my results with Stefan's graph [2]. And the fact is that with a very low shared_buffers configuration, my results are quite similar to Stefan's results but, as soon as I put higher values of shared_buffers, performances begins degrading [3]. I performed my tests with: pgbench -i -s 100 -U postgres bench and pgbench -s 100 -c 100 -t 3 -U postgres bench. Of course, I initialize the database before each run. I made my tests in one direction then in the other with similar results so it's not a degradation due to consecutive runs. I lowered the number of concurrent clients to 50 because 100 is quite high and I obtain the same sort of results: shared_buffers=32MB: 1869 tps shared_buffers=64MB: 1844 tps shared_buffers=512MB: 1676 tps shared_buffers=1024MB: 1559 tps Non default parameters are: max_connections = 200 work_mem = 32MB wal_buffers = 1024kB checkpoint_segments = 192 effective_cache_size = 5GB (I use more or less the configuration used by Stefan - I had the same behaviour with default wal_buffers and checkpoint_segments) While monitoring the server with vmstat, I can't see any real reason why it's slower. When shared_buffers has a higher value, I/O are lower, context switches too and finally performances. The CPU usage is quite similar (~50-60%). I/O doesn't limit the performances AFAICS. I must admit I'm a bit puzzled. Does anyone have any pointer which could explain this behaviour or any way to track the issue? I'll be glad to perform any test needed to understand the problem. Thanks. [1] http://www.kaltenbrunner.cc/blog/index.php?/archives/21-8.3-vs.-8.2-a-simple-benchmark.html [2] http://www.kaltenbrunner.cc/blog/uploads/83b4shm.gif [3] http://people.openwide.fr/~gsmet/postgresql/tps_shared_buffers.png (X=shared_buffers in MB/Y=results with pgbench) -- Guillaume ---(end of broadcast)--- TIP 6: explain analyze is your friend
