Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-19 Thread Greg Smith
On 09/05/2011 07:52 PM, Tomas Vondra wrote: If your logging criteria for the write phase was display a message any time more than 30 seconds have passed since last seeing one, that would give you only a few lines of output in a boring, normal checkpoint--certainly less than the 9 in-progress

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-19 Thread Tom Lane
Greg Smith g...@2ndquadrant.com writes: One thing I am trying to avoid here is needing to check the system clock after every buffer write. On machines where gettimeofday is slow (and last I heard there were still lots of them), any such thing would be a disaster performance-wise. I'm still

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-19 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: On machines where gettimeofday is slow (and last I heard there were still lots of them), any such thing would be a disaster performance-wise. I'm still afraid to add more gettimeofday's into the query parse/plan/execute code path, even though it would

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-05 Thread Tomas Vondra
On 2 Září 2011, 22:04, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: On 2 Z?? 2011, 21:23, Tom Lane wrote: Well, to be blunt, putting stuff into the postmaster log is entirely the wrong way to satify a requirement like that. If you want to expose progress information, it should be

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-05 Thread Tomas Vondra
On 3 Září 2011, 8:19, Greg Smith wrote: If you're expanding log_checkpoints to an enum, for that to handle what I think everybody might ever want (for what checkpoints do now at least), I'd find that more useful if it happened like this instead: log_checkpoints = {off, on, write, sync,

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-05 Thread Robert Haas
On Mon, Sep 5, 2011 at 7:52 PM, Tomas Vondra t...@fuzzy.cz wrote: If your logging criteria for the write phase was display a message any time more than 30 seconds have passed since last seeing one, that would give you only a few lines of output in a boring, normal checkpoint--certainly less

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-03 Thread Greg Smith
On 09/02/2011 11:10 AM, Tomas Vondra wrote: My 'ideal' solution would be either to add another GUC (to turn this on/off) or allow log_checkpoints to have three values log_checkpoints = {off, normal, detailed} where 'normal' provides the current output and 'detail' produces this much verbose

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Magnus Hagander
On Thu, Sep 1, 2011 at 21:59, Tomas Vondra t...@fuzzy.cz wrote: I've prepared a significantly simplified version of the patch. The two main changes are (a) it does not update the pg_stat_bgwriter anymore, it just prints an info to the server log (b) a new GUC is not required, it's driven by

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tomas Vondra
On 2 Září 2011, 9:47, Magnus Hagander wrote: On Thu, Sep 1, 2011 at 21:59, Tomas Vondra t...@fuzzy.cz wrote: I've prepared a significantly simplified version of the patch. The two main changes are (a) it does not update the pg_stat_bgwriter anymore, it just prints an info to the server log

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Magnus Hagander
On Fri, Sep 2, 2011 at 11:01, Tomas Vondra t...@fuzzy.cz wrote: On 2 Září 2011, 9:47, Magnus Hagander wrote: On Thu, Sep 1, 2011 at 21:59, Tomas Vondra t...@fuzzy.cz wrote: I've prepared a significantly simplified version of the patch. The two main changes are (a) it does not update the

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tomas Vondra
On 2 Září 2011, 12:45, Magnus Hagander wrote: On Fri, Sep 2, 2011 at 11:01, Tomas Vondra t...@fuzzy.cz wrote: What about logging it with a lower level, e.g. NOTICE instead of the current LOG? If that's not a solution then a new GUC is needed I guess. I guess if it's at a DEBUG level it won't

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Peter Eisentraut
On fre, 2011-09-02 at 11:01 +0200, Tomas Vondra wrote: What about logging it with a lower level, e.g. NOTICE instead of the current LOG? If that's not a solution then a new GUC is needed I guess. Changing the log level is not the appropriate solution. Make it a configuration parameter. --

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On fre, 2011-09-02 at 11:01 +0200, Tomas Vondra wrote: What about logging it with a lower level, e.g. NOTICE instead of the current LOG? If that's not a solution then a new GUC is needed I guess. Changing the log level is not the appropriate solution.

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tomas Vondra
On 2 Září 2011, 15:44, Peter Eisentraut wrote: On fre, 2011-09-02 at 11:01 +0200, Tomas Vondra wrote: What about logging it with a lower level, e.g. NOTICE instead of the current LOG? If that's not a solution then a new GUC is needed I guess. Changing the log level is not the appropriate

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Magnus Hagander
On Sep 2, 2011 5:02 PM, Tomas Vondra t...@fuzzy.cz wrote: On 2 Září 2011, 15:44, Peter Eisentraut wrote: On fre, 2011-09-02 at 11:01 +0200, Tomas Vondra wrote: What about logging it with a lower level, e.g. NOTICE instead of the current LOG? If that's not a solution then a new GUC is

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tomas Vondra
On 2 Září 2011, 16:13, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On fre, 2011-09-02 at 11:01 +0200, Tomas Vondra wrote: What about logging it with a lower level, e.g. NOTICE instead of the current LOG? If that's not a solution then a new GUC is needed I guess. Changing the

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tomas Vondra
On 2 Září 2011, 17:08, Magnus Hagander wrote: On Sep 2, 2011 5:02 PM, Tomas Vondra t...@fuzzy.cz wrote: On 2 Září 2011, 15:44, Peter Eisentraut wrote: On fre, 2011-09-02 at 11:01 +0200, Tomas Vondra wrote: What about logging it with a lower level, e.g. NOTICE instead of the current LOG?

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Kevin Grittner
.Tomas Vondra t...@fuzzy.cz wrote: On 2 Zá*í 2011, 15:44, Peter Eisentraut wrote: Changing the log level is not the appropriate solution. Make it a configuration parameter. Why is it inappropriate solution? There's a log_checkpoints GUC that drives it and you can either get basic info

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Magnus Hagander
On Sep 2, 2011 5:18 PM, Jaime Casanova ja...@2ndquadrant.com wrote: On Fri, Sep 2, 2011 at 10:10 AM, Tomas Vondra t...@fuzzy.cz wrote: My 'ideal' solution would be either to add another GUC (to turn this on/off) or allow log_checkpoints to have three values log_checkpoints = {off,

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Jaime Casanova
On Fri, Sep 2, 2011 at 10:10 AM, Tomas Vondra t...@fuzzy.cz wrote: My 'ideal' solution would be either to add another GUC (to turn this on/off) or allow log_checkpoints to have three values log_checkpoints = {off, normal, detailed} +1 on make it an enum but i prefer values {on, off,

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Robert Haas
On Thu, Sep 1, 2011 at 3:59 PM, Tomas Vondra t...@fuzzy.cz wrote: I've prepared a significantly simplified version of the patch. The two main changes are (a) it does not update the pg_stat_bgwriter anymore, it just prints an info to the server log (b) a new GUC is not required, it's driven

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Peter Eisentraut
On fre, 2011-09-02 at 17:02 +0200, Tomas Vondra wrote: On 2 Září 2011, 15:44, Peter Eisentraut wrote: On fre, 2011-09-02 at 11:01 +0200, Tomas Vondra wrote: What about logging it with a lower level, e.g. NOTICE instead of the current LOG? If that's not a solution then a new GUC is needed I

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On fre, 2011-09-02 at 17:02 +0200, Tomas Vondra wrote: Why is it inappropriate solution? There's a log_checkpoints GUC that drives it and you can either get basic info (summary of the checkpoint) or detailed log (with a lower log level). If a user is

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tomas Vondra
On 2 Září 2011, 20:46, Peter Eisentraut wrote: On fre, 2011-09-02 at 17:13 +0200, Tomas Vondra wrote: Sure, but I think the effort not to have a zillion of GUC makes sense. Well, I'll be the first to agree that reducing complexity in configuration and tuning settings is worth aiming for.

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tomas Vondra
On 2 Září 2011, 20:48, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On fre, 2011-09-02 at 17:02 +0200, Tomas Vondra wrote: Why is it inappropriate solution? There's a log_checkpoints GUC that drives it and you can either get basic info (summary of the checkpoint) or detailed log

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tom Lane
Tomas Vondra t...@fuzzy.cz writes: On 2 Září 2011, 20:48, Tom Lane wrote: Yeah, if we're going to have this at all, some form of GUC control over it seems necessary. I'm still disturbed by the verbosity of the proposed output though. Couldn't we collapse the information into a single log

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tomas Vondra
On 2 Září 2011, 21:23, Tom Lane wrote: Tomas Vondra t...@fuzzy.cz writes: On 2 Z?? 2011, 20:48, Tom Lane wrote: Yeah, if we're going to have this at all, some form of GUC control over it seems necessary. I'm still disturbed by the verbosity of the proposed output though. Couldn't we

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-02 Thread Tom Lane
Tomas Vondra t...@fuzzy.cz writes: On 2 Září 2011, 21:23, Tom Lane wrote: Well, to be blunt, putting stuff into the postmaster log is entirely the wrong way to satify a requirement like that. If you want to expose progress information, it should be exposed via something dynamic like

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-09-01 Thread Tomas Vondra
I've prepared a significantly simplified version of the patch. The two main changes are (a) it does not update the pg_stat_bgwriter anymore, it just prints an info to the server log (b) a new GUC is not required, it's driven by the log_checkpoints This version will log at least 10 'checkpoint

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-29 Thread Tomas Vondra
On 29 Srpen 2011, 7:47, Noah Misch wrote: On Sat, Aug 27, 2011 at 03:57:16PM +0200, Tomas Vondra wrote: On 27 Srpen 2011, 6:01, Noah Misch wrote: Could you remove this hazard by adding a step 2a. psql -c CHECKPOINT? I already do that, but it really does not solve the issue. It just aligns

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-28 Thread Noah Misch
On Sat, Aug 27, 2011 at 03:57:16PM +0200, Tomas Vondra wrote: On 27 Srpen 2011, 6:01, Noah Misch wrote: On Fri, Aug 26, 2011 at 10:46:33AM +0200, Tomas Vondra wrote: 1. collect pg_stat_bgwriter stats 2. run pgbench for 10 minutes 3. collect pg_stat_bgwriter stats (to compute difference

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-27 Thread Greg Smith
On 08/27/2011 12:01 AM, Noah Misch wrote: On Fri, Aug 26, 2011 at 10:46:33AM +0200, Tomas Vondra wrote: 1. collect pg_stat_bgwriter stats 2. run pgbench for 10 minutes 3. collect pg_stat_bgwriter stats (to compute difference with (1)) 4. kill the postmaster The problem is that when

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-27 Thread Tomas Vondra
On 27 Srpen 2011, 6:01, Noah Misch wrote: On Fri, Aug 26, 2011 at 10:46:33AM +0200, Tomas Vondra wrote: 1. collect pg_stat_bgwriter stats 2. run pgbench for 10 minutes 3. collect pg_stat_bgwriter stats (to compute difference with (1)) 4. kill the postmaster The problem is that when

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-26 Thread Greg Smith
On 08/25/2011 04:57 PM, Tomas Vondra wrote: (b) sends bgwriter stats (so that the buffers_checkpoint is updated) The idea behind only updating the stats in one chunk, at the end, is that it makes one specific thing easier to do. Let's say you're running a monitoring system that is

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-26 Thread Magnus Hagander
On Thu, Aug 25, 2011 at 22:57, Tomas Vondra t...@fuzzy.cz wrote: Hello, I'd like to propose a small patch that allows better checkpoint progress monitoring. The patch is quite simple - it adds a new integer GUC checkpoint_update_limit and every time checkpoint writes this number of buffers,

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-26 Thread Tomas Vondra
On 26 Srpen 2011, 9:35, Greg Smith wrote: On 08/25/2011 04:57 PM, Tomas Vondra wrote: (b) sends bgwriter stats (so that the buffers_checkpoint is updated) As for throwing more log data out, I'm not sure what new analysis you're thinking of that it allows. I/O gets increasingly spiky as you

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-26 Thread Greg Smith
On 08/26/2011 03:54 AM, Magnus Hagander wrote: In theory, this could be the progress view or progress field talked about around Gregs previous patch - or it could just be modifying the commandstring in pg_stat_activity. Right. The whole progress indicator idea is hard to do for queries in

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-26 Thread Tomas Vondra
On 26 Srpen 2011, 9:54, Magnus Hagander wrote: This seems like the wrong thing to write to the log. It's really only useful *during* the checkpoint run, isn't it? If so, I think it should go in a pg_stat view. In theory, this could be the progress view or progress field talked about around

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-26 Thread Tomas Vondra
On 26 Srpen 2011, 19:17, Greg Smith wrote: On 08/26/2011 03:54 AM, Magnus Hagander wrote: In theory, this could be the progress view or progress field talked about around Gregs previous patch - or it could just be modifying the commandstring in pg_stat_activity. Right. The whole progress

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-26 Thread Noah Misch
On Fri, Aug 26, 2011 at 10:46:33AM +0200, Tomas Vondra wrote: Hmmm, let me explain what led me to this patch - right now I'm doing a comparison of filesystems with various block sizes (both fs and db blocks). I've realized that the db block size significantly influences frequency of

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-25 Thread Josh Berkus
Tomas, I'd like to propose a small patch that allows better checkpoint progress monitoring. The patch is quite simple - it adds a new integer GUC checkpoint_update_limit and every time checkpoint writes this number of buffers, it does two things: I'd rather not have a new GUC if we can avoid

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-25 Thread Tomas Vondra
On 26 Srpen 2011, 0:18, Josh Berkus wrote: Tomas, I'd like to propose a small patch that allows better checkpoint progress monitoring. The patch is quite simple - it adds a new integer GUC checkpoint_update_limit and every time checkpoint writes this number of buffers, it does two things:

Re: [HACKERS] PATCH: regular logging of checkpoint progress

2011-08-25 Thread Tomas Vondra
On 26 Srpen 2011, 0:39, Tomas Vondra wrote: On 26 Srpen 2011, 0:18, Josh Berkus wrote: Tomas, I'd like to propose a small patch that allows better checkpoint progress monitoring. The patch is quite simple - it adds a new integer GUC checkpoint_update_limit and every time checkpoint writes