Re: [HACKERS] checkpoint patches

2012-04-06 Thread Greg Smith
On 04/05/2012 02:23 PM, Jim Nasby wrote: If there's a fundamental flaw in how linux deals with heavy writes that means you can't rely on certain latency windows, perhaps we should be looking at using a different OS to test those cases... Performance under this sort of write overload is

Re: [HACKERS] checkpoint patches

2012-04-05 Thread Jim Nasby
On 4/3/12 11:30 PM, Greg Smith wrote: On 03/25/2012 04:29 PM, Jim Nasby wrote: Another $0.02: I don't recall the community using pg_bench much at all to measure latency... I believe it's something fairly new. I point this out because I believe there are differences in analysis that you need

Re: [HACKERS] checkpoint patches

2012-04-03 Thread Greg Smith
On 03/25/2012 04:29 PM, Jim Nasby wrote: Another $0.02: I don't recall the community using pg_bench much at all to measure latency... I believe it's something fairly new. I point this out because I believe there are differences in analysis that you need to do for TPS vs latency. I think

Re: [HACKERS] checkpoint patches

2012-03-26 Thread Robert Haas
On Sun, Mar 25, 2012 at 4:29 PM, Jim Nasby j...@nasby.net wrote: I wouldn't be too quick to dismiss increasing checkpoint frequency (ie: decreasing checkpoint_timeout). I'm not dismissing that, but my tests show only a very small gain in that area. Now there may be another test where it shows

Re: [HACKERS] checkpoint patches

2012-03-25 Thread Jim Nasby
On 3/23/12 7:38 AM, Robert Haas wrote: And here are the latency results for 95th-100th percentile with checkpoint_timeout=16min. ckpt.master.13: 1703, 1830, 2166, 17953, 192434, 43946669 ckpt.master.14: 1728, 1858, 2169, 15596, 187943, 9619191 ckpt.master.15: 1700, 1835, 2189, 22181, 206445,

Re: [HACKERS] checkpoint patches

2012-03-23 Thread Robert Haas
On Thu, Mar 22, 2012 at 8:44 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: On Thu, Mar 22, 2012 at 3:45 PM, Stephen Frost sfr...@snowman.net wrote: Well, those numbers just aren't that exciting. :/ Agreed.  There's clearly an effect, but on this

Re: [HACKERS] checkpoint patches

2012-03-23 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: Well, how do you want to look at it? I thought the last graph you provided was a useful way to view the results. It was my intent to make that clear in my prior email, my apologies if that didn't come through. Here's the data from 80th

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Robert Haas
On Wed, Mar 21, 2012 at 3:38 PM, Robert Haas robertmh...@gmail.com wrote: It looks like I neglected to record that information for the last set of runs.  But I can try another set of runs and gather that information. OK. On further review, my previous test script contained several bugs. So

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Robert Haas
On Thu, Mar 22, 2012 at 9:07 AM, Robert Haas robertmh...@gmail.com wrote: However, looking at this a bit more, I think the checkpoint-sync-pause-v1 patch contains an obvious bug - the GUC is supposedly represented in seconds (though not marked with GUC_UNIT_S, oops) but what the sleep

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: TPS numbers: checkpoint-sync-pause-v1: 2594.448538, 2600.231666, 2580.041376 master: 2466.31, 2450.752843, 2291.613305 90th percentile latency: checkpoint-sync-pause-v1: 1487, 1488, 1481 master: 1493, 1519, 1507 Well, those numbers just

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Robert Haas
On Thu, Mar 22, 2012 at 3:45 PM, Stephen Frost sfr...@snowman.net wrote: Well, those numbers just aren't that exciting. :/ Agreed. There's clearly an effect, but on this test it's not very big. Then again, I'm a bit surprised that the latencies aren't worse, perhaps the previous improvements

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Jeff Janes
On Thu, Mar 22, 2012 at 6:07 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Mar 21, 2012 at 3:38 PM, Robert Haas robertmh...@gmail.com wrote: It looks like I neglected to record that information for the last set of runs.  But I can try another set of runs and gather that information.

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Robert Haas
On Thu, Mar 22, 2012 at 7:03 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Thu, Mar 22, 2012 at 6:07 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Mar 21, 2012 at 3:38 PM, Robert Haas robertmh...@gmail.com wrote: It looks like I neglected to record that information for the last set of

Re: [HACKERS] checkpoint patches

2012-03-22 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Thu, Mar 22, 2012 at 3:45 PM, Stephen Frost sfr...@snowman.net wrote: Well, those numbers just aren't that exciting. :/ Agreed. There's clearly an effect, but on this test it's not very big. Ok, perhaps that was because of how you were

[HACKERS] checkpoint patches

2012-03-21 Thread Robert Haas
There are two checkpoint-related patches in this CommitFest that haven't gotten much love, one from me and the other from Greg Smith: https://commitfest.postgresql.org/action/patch_view?id=752 https://commitfest.postgresql.org/action/patch_view?id=795 Mine uses sync_file_range() when available

Re: [HACKERS] checkpoint patches

2012-03-21 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: Thoughts? It was my impression that these patches were much about improving overall tps and more about reducing latency spikes for specific transactions that get hit by a checkpoint happening at a bad time. Are there any reductions in max

Re: [HACKERS] checkpoint patches

2012-03-21 Thread Robert Haas
On Wed, Mar 21, 2012 at 3:34 PM, Stephen Frost sfr...@snowman.net wrote: Robert, * Robert Haas (robertmh...@gmail.com) wrote: Thoughts? It was my impression that these patches were much about improving overall tps and more about reducing latency spikes for specific transactions that get