Re: [PERFORM] Tuning Postgres 9.1 on Windows
Exactly, if turning off fsync gives me 100 commits/sec then I know where my bottleneck is and I can attack it. Keep in mind though that I already turned off synchronous commit -- *really* dangerous -- and it didn't have any effect. -- Les -Original Message- From: pgsql-performance-ow...@postgresql.org [mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Merlin Moncure Sent: Monday, April 30, 2012 6:04 PM To: Thomas Kellerer Cc: pgsql-performance@postgresql.org Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows On Mon, Apr 30, 2012 at 5:00 PM, Thomas Kellerer spam_ea...@gmx.net wrote: Merlin Moncure wrote on 30.04.2012 23:43: Trying turning off fsync in postgrsql.conf to be sure. This is a dangerous advise. Turning off fsync can potentially corrupt the database in case of a system failure (e.g. power outage). sure. that said, we're just trying to figure out why he's getting around 40tps. since he's only benchmarking test data it's perfectly ok to do that. merlin -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are confidential. If you are not the named recipient please notify the sender and immediately delete it. You may not disseminate, distribute, or forward this e-mail message or disclose its contents to anybody else. Copyright and any other intellectual property rights in its contents are the sole property of Cantor Fitzgerald. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Although we routinely screen for viruses, addressees should check this e-mail and any attachments for viruses. We make no representation or warranty as to the absence of viruses in this e-mail or any attachments. Please note that to ensure regulatory compliance and for the protection of our customers and business, we may monitor and read e-mails sent to and from our server(s). For further important information, please see http://www.cantor.com/legal/statement -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Tuning Postgres 9.1 on Windows
On Tue, May 1, 2012 at 7:51 AM, Walker, James Les jawal...@cantor.com wrote: Exactly, if turning off fsync gives me 100 commits/sec then I know where my bottleneck is and I can attack it. Keep in mind though that I already turned off synchronous commit -- *really* dangerous -- and it didn't have any effect. well synchronous commit is not as dangerous: fsync off + power failure = corrupt database synchronous commit off + power failure = some lost transactions still waiting on the ssd model #. worst case scenario is that you tps rate is in fact sync bound and you have a ssd without capacitor backed buffers (for example, the intel 320 has them); the probable workaround would be to set the drive cache from write through to write back but it would unsafe in that case. in other words, tps rates in the triple digits would be physically impossible. another less likely scenario is you are having network issues (assuming you are connecting to the database through tcp/ip). 20 years in, microsoft is still figuring out how to properly configure a network socket. merlin -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Tuning Postgres 9.1 on Windows
SSD is OCZ-VERTEX3 MI. Controller is LSI SAS2 2008 Falcon. I'm working on installing EDB. Then I can give you some I/O numbers. -- Les -Original Message- From: Merlin Moncure [mailto:mmonc...@gmail.com] Sent: Tuesday, May 01, 2012 9:07 AM To: Walker, James Les Cc: Thomas Kellerer; pgsql-performance@postgresql.org Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows On Tue, May 1, 2012 at 7:51 AM, Walker, James Les jawal...@cantor.com wrote: Exactly, if turning off fsync gives me 100 commits/sec then I know where my bottleneck is and I can attack it. Keep in mind though that I already turned off synchronous commit -- *really* dangerous -- and it didn't have any effect. well synchronous commit is not as dangerous: fsync off + power failure = corrupt database synchronous commit off + power failure = some lost transactions still waiting on the ssd model #. worst case scenario is that you tps rate is in fact sync bound and you have a ssd without capacitor backed buffers (for example, the intel 320 has them); the probable workaround would be to set the drive cache from write through to write back but it would unsafe in that case. in other words, tps rates in the triple digits would be physically impossible. another less likely scenario is you are having network issues (assuming you are connecting to the database through tcp/ip). 20 years in, microsoft is still figuring out how to properly configure a network socket. merlin CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are confidential. If you are not the named recipient please notify the sender and immediately delete it. You may not disseminate, distribute, or forward this e-mail message or disclose its contents to anybody else. Copyright and any other intellectual property rights in its contents are the sole property of Cantor Fitzgerald. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Although we routinely screen for viruses, addressees should check this e-mail and any attachments for viruses. We make no representation or warranty as to the absence of viruses in this e-mail or any attachments. Please note that to ensure regulatory compliance and for the protection of our customers and business, we may monitor and read e-mails sent to and from our server(s). For further important information, please see http://www.cantor.com/legal/statement -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Tuning Postgres 9.1 on Windows
On 5/1/2012 8:06 AM, Merlin Moncure wrote: On Tue, May 1, 2012 at 7:51 AM, Walker, James Lesjawal...@cantor.com wrote: Exactly, if turning off fsync gives me 100 commits/sec then I know where my bottleneck is and I can attack it. Keep in mind though that I already turned off synchronous commit -- *really* dangerous -- and it didn't have any effect. well synchronous commit is not as dangerous: fsync off + power failure = corrupt database synchronous commit off + power failure = some lost transactions still waiting on the ssd model #. worst case scenario is that you tps rate is in fact sync bound and you have a ssd without capacitor backed buffers (for example, the intel 320 has them); the probable workaround would be to set the drive cache from write through to write back but it would unsafe in that case. in other words, tps rates in the triple digits would be physically impossible. another less likely scenario is you are having network issues (assuming you are connecting to the database through tcp/ip). 20 years in, microsoft is still figuring out how to properly configure a network socket. merlin Even if its all local, windows doesnt have domain sockets (correct?), so all that traffic still has to go thru some bit of network stack, yes? -Andy -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Tuning Postgres 9.1 on Windows
On Tue, May 1, 2012 at 8:14 AM, Walker, James Les jawal...@cantor.com wrote: SSD is OCZ-VERTEX3 MI. Controller is LSI SAS2 2008 Falcon. I'm working on installing EDB. Then I can give you some I/O numbers. It looks like the ssd doesn't have a nv cache and the raid card is a simple sas hba (which likely isn't doing much for the ssd besides masking TRIM). The OCZ 'pro' versions are the ones with power loss protection (see: http://hothardware.com/Reviews/OCZ-Vertex-3-Pro-SandForce-SF2000-Based-SSD-Preview/). Note the bullet: Implements SandForce 2582 Controller with power loss data protection. It doesn't look like the Vertex 3 Pro is out yet. If my hunch is correct, the issue here is that the drive is being asked to sync data physically and SSD really don't perform well when the controller isn't in control of when and how to sync data. However full physical sync is the only way to guarantee data is truly safe in the context of a unexpected power loss (an nv cache is basically a compromise on this point). merlin -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
[PERFORM] Any disadvantages of using =ANY(ARRAY()) instead of IN?
Hi, I am using postgresql as database for a hibernate based java oltp project and as in previous projects am totally impressed by postgresql's robustness, performance and feature-richness. Thanks for this excellent piece of software. Quite often Hibernate ends up generating queries with a lot of joins which usually works well, except for queries which load some additional data based on a previous query (SUBSELECT collections), which look like: select . from table1 ... left outer join table 15 WHERE table1.id IN (select id join table16 ... join table20 WHERE table20.somevalue=?) Starting with some amount of joins, the optimizer starts to do quite suboptimal things like hash-joining huge tables where selctivity would very low. I already raised join_collapse_limit and from_collapse_limit, but after a certain point query planning starts to become very expensive. However, when using =ANY(ARRAY(select ...)) instead of IN the planner seems to do a lot better, most likely because it treats the subquery as a black-box that needs to be executed independently. I've hacked hibernate a bit to use ANY+ARRAY, and it seems to work a lot better than using IN. However, I am a bit uncertain: - Is it safe to use ANY(ARRAY(select ...)) when I know the sub-query will only return a small amount (0-100s) of rows? - Shouldn't the optimizer be a bit smarter avoiding optimizing this case in the first place, instead of bailing out later? Should I file a bug-report about this problem? Thank you in advance, Clemens -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Any disadvantages of using =ANY(ARRAY()) instead of IN?
Clemens Eisserer linuxhi...@gmail.com writes: Quite often Hibernate ends up generating queries with a lot of joins which usually works well, except for queries which load some additional data based on a previous query (SUBSELECT collections), which look like: select . from table1 ... left outer join table 15 WHERE table1.id IN (select id join table16 ... join table20 WHERE table20.somevalue=?) Starting with some amount of joins, the optimizer starts to do quite suboptimal things like hash-joining huge tables where selctivity would very low. I already raised join_collapse_limit and from_collapse_limit, but after a certain point query planning starts to become very expensive. What PG version are we talking about here? However, when using =ANY(ARRAY(select ...)) instead of IN the planner seems to do a lot better, most likely because it treats the subquery as a black-box that needs to be executed independently. I've hacked hibernate a bit to use ANY+ARRAY, and it seems to work a lot better than using IN. That doesn't sound like a tremendously good idea to me. But with so few details, it's hard to comment intelligently. Can you provide a concrete test case? regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Tuning Postgres 9.1 on Windows
I installed the enterprisedb distribution and immediately saw a 400% performance increase. Turning off fsck made it an order of magnitude better. I'm now peaking at over 400 commits per second. Does that sound right? If I understand what you're saying, then to sustain this high rate I'm going to need a controller that can defer fsync requests from the host because it has some sort of battery backup that guarantees the full write. -- Les -Original Message- From: Merlin Moncure [mailto:mmonc...@gmail.com] Sent: Tuesday, May 01, 2012 9:43 AM To: Walker, James Les Cc: Thomas Kellerer; pgsql-performance@postgresql.org Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows On Tue, May 1, 2012 at 8:14 AM, Walker, James Les jawal...@cantor.com wrote: SSD is OCZ-VERTEX3 MI. Controller is LSI SAS2 2008 Falcon. I'm working on installing EDB. Then I can give you some I/O numbers. It looks like the ssd doesn't have a nv cache and the raid card is a simple sas hba (which likely isn't doing much for the ssd besides masking TRIM). The OCZ 'pro' versions are the ones with power loss protection (see: http://hothardware.com/Reviews/OCZ-Vertex-3-Pro-SandForce-SF2000-Based-SSD-Preview/). Note the bullet: Implements SandForce 2582 Controller with power loss data protection. It doesn't look like the Vertex 3 Pro is out yet. If my hunch is correct, the issue here is that the drive is being asked to sync data physically and SSD really don't perform well when the controller isn't in control of when and how to sync data. However full physical sync is the only way to guarantee data is truly safe in the context of a unexpected power loss (an nv cache is basically a compromise on this point). merlin CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are confidential. If you are not the named recipient please notify the sender and immediately delete it. You may not disseminate, distribute, or forward this e-mail message or disclose its contents to anybody else. Copyright and any other intellectual property rights in its contents are the sole property of Cantor Fitzgerald. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Although we routinely screen for viruses, addressees should check this e-mail and any attachments for viruses. We make no representation or warranty as to the absence of viruses in this e-mail or any attachments. Please note that to ensure regulatory compliance and for the protection of our customers and business, we may monitor and read e-mails sent to and from our server(s). For further important information, please see http://www.cantor.com/legal/statement -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Any disadvantages of using =ANY(ARRAY()) instead of IN?
Hi Tom, Thanks for your reply. What PG version are we talking about here? For development I use 9.1.3, on the production server is 8.4.7 - happens with both cases. That doesn't sound like a tremendously good idea to me. Could you elaborate on the downsides of this approach a bit? But with so few details, it's hard to comment intelligently. Can you provide a concrete test case? A self contained testcase would take some time to create (and list members willing to configure and run), so I hope a query as well as an explain-analyze run will provide more information (done with 9.1.3): http://pastebin.com/BGRdAPg2 Its kind of the worst-worst case which I will improve later (way too much relations loaded through join-fetching), but its quite a good way to show the issue. Replacing the IN with a ANY(ARRAY()) already yields a way better plan. Thank you in advance, Clemens -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Tuning Postgres 9.1 on Windows
On Tue, May 1, 2012 at 9:44 AM, Walker, James Les jawal...@cantor.com wrote: I installed the enterprisedb distribution and immediately saw a 400% performance increase. Turning off fsck made it an order of magnitude better. I'm now peaking at over 400 commits per second. Does that sound right? yeah -- well it's hard to say but that sounds plausible based on what i know. it would be helpful to see the queries you're running to get apples to apples idea of what's going on. If I understand what you're saying, then to sustain this high rate I'm going to need a controller that can defer fsync requests from the host because it has some sort of battery backup that guarantees the full write. yes -- historically, they way to get your tps rate up was to get a battery backed cache. this can give you burst (although not necessarily sustained) tps rates well above what the drive can handle. lately, a few of the better built ssd also have on board capacitors which provide a similar function and allow the drives to safely hit high tps rates as well. take a good look at the intel 320 and 710 drives. merlin -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Tuning Postgres 9.1 on Windows
Walker, James Les wrote on 01.05.2012 16:44: I installed the enterprisedb distribution and immediately saw a 400% performance increase. What exactly is the enterprisedb distribution? Are you talking about the the Advanced Server? I would be very surprised if the code base would differ so much to allow such a performance gain. Could it be that the default settings for the Advanced Server are different than those of the community edition? And what did you have installed before that? (as the Windows binary are always distributed by EnterpriseDB) Thomas -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Tuning Postgres 9.1 on Windows
Yes. I didn't know the proper vernacular :-) It is very likely that the default settings are different. I'm looking at that right now. -- Les Walker -Original Message- From: pgsql-performance-ow...@postgresql.org [mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Thomas Kellerer Sent: Tuesday, May 01, 2012 1:00 PM To: pgsql-performance@postgresql.org Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows Walker, James Les wrote on 01.05.2012 16:44: I installed the enterprisedb distribution and immediately saw a 400% performance increase. What exactly is the enterprisedb distribution? Are you talking about the the Advanced Server? I would be very surprised if the code base would differ so much to allow such a performance gain. Could it be that the default settings for the Advanced Server are different than those of the community edition? And what did you have installed before that? (as the Windows binary are always distributed by EnterpriseDB) Thomas -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are confidential. If you are not the named recipient please notify the sender and immediately delete it. You may not disseminate, distribute, or forward this e-mail message or disclose its contents to anybody else. Copyright and any other intellectual property rights in its contents are the sole property of Cantor Fitzgerald. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Although we routinely screen for viruses, addressees should check this e-mail and any attachments for viruses. We make no representation or warranty as to the absence of viruses in this e-mail or any attachments. Please note that to ensure regulatory compliance and for the protection of our customers and business, we may monitor and read e-mails sent to and from our server(s). For further important information, please see http://www.cantor.com/legal/statement -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Tuning Postgres 9.1 on Windows
Turns out the 40% was due to a configuration problem with my application. I'm now getting the same performance with community edition. It appears that I'm now CPU bound. My CPU's are all pegged. -- Les Walker -Original Message- From: pgsql-performance-ow...@postgresql.org [mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Walker, James Les Sent: Tuesday, May 01, 2012 3:14 PM To: 'Thomas Kellerer'; pgsql-performance@postgresql.org Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows Yes. I didn't know the proper vernacular :-) It is very likely that the default settings are different. I'm looking at that right now. -- Les Walker -Original Message- From: pgsql-performance-ow...@postgresql.org [mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Thomas Kellerer Sent: Tuesday, May 01, 2012 1:00 PM To: pgsql-performance@postgresql.org Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows Walker, James Les wrote on 01.05.2012 16:44: I installed the enterprisedb distribution and immediately saw a 400% performance increase. What exactly is the enterprisedb distribution? Are you talking about the the Advanced Server? I would be very surprised if the code base would differ so much to allow such a performance gain. Could it be that the default settings for the Advanced Server are different than those of the community edition? And what did you have installed before that? (as the Windows binary are always distributed by EnterpriseDB) Thomas -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are confidential. If you are not the named recipient please notify the sender and immediately delete it. You may not disseminate, distribute, or forward this e-mail message or disclose its contents to anybody else. Copyright and any other intellectual property rights in its contents are the sole property of Cantor Fitzgerald. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Although we routinely screen for viruses, addressees should check this e-mail and any attachments for viruses. We make no representation or warranty as to the absence of viruses in this e-mail or any attachments. Please note that to ensure regulatory compliance and for the protection of our customers and business, we may monitor and read e-mails sent to and from our server(s). For further important information, please see http://www.cantor.com/legal/statement -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
