Re: [HACKERS] Instrument checkpoint sync calls

2010-12-16 Thread Greg Smith
Robert Haas wrote: On Wed, Dec 15, 2010 at 9:22 AM, Greg Smith g...@2ndquadrant.com wrote: patch I submit. Doesn't seem worth going through the trouble of committing that minor rework on its own, I'll slip it into the next useful thing that touches this area I do. Thanks for the hint, this

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-16 Thread Robert Haas
On Thu, Dec 16, 2010 at 6:48 AM, Greg Smith g...@2ndquadrant.com wrote: I meant that I'd bundle it into the block of time I spend on that, and likely submit with something else that touches the same area.  Obviously the correction patch would be better on its own when being handed over to a

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-15 Thread Greg Smith
Alvaro Herrera wrote: I gave this patch a look and it seems pretty good to me, except that I'm uncomfortable with the idea of mdsync filling in the details for CheckpointStats fields directly. Would it work to pass a struct (say SmgrSyncStats) from CheckPointBuffers to smgrsync and from there

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-15 Thread Robert Haas
On Wed, Dec 15, 2010 at 9:22 AM, Greg Smith g...@2ndquadrant.com wrote: patch I submit.  Doesn't seem worth going through the trouble of committing that minor rework on its own, I'll slip it into the next useful thing that touches this area I do.  Thanks for the hint, this would work better

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-14 Thread Greg Smith
Robert Haas wrote: I took a look at this and it looks generally good, but I'm wondering why md.c is converting the results from an exact value to a floating point, only to have xlog.c turn around and convert back to an integer. I think it could just return milliseconds directly, or if you're

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-14 Thread Alvaro Herrera
I gave this patch a look and it seems pretty good to me, except that I'm uncomfortable with the idea of mdsync filling in the details for CheckpointStats fields directly. Would it work to pass a struct (say SmgrSyncStats) from CheckPointBuffers to smgrsync and from there to mdsync, have this

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-14 Thread Robert Haas
On Tue, Dec 14, 2010 at 9:29 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: I gave this patch a look and it seems pretty good to me, except Err, woops. I just committed this as-is. Sorry. that I'm uncomfortable with the idea of mdsync filling in the details for CheckpointStats fields

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-14 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar dic 14 11:34:55 -0300 2010: On Tue, Dec 14, 2010 at 9:29 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: I gave this patch a look and it seems pretty good to me, except Err, woops. I just committed this as-is. Sorry. I noticed :-) that

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-14 Thread Robert Haas
On Tue, Dec 14, 2010 at 11:47 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: that I'm uncomfortable with the idea of mdsync filling in the details for CheckpointStats fields directly.  Would it work to pass a struct (say SmgrSyncStats) from CheckPointBuffers to smgrsync and from

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-13 Thread Greg Smith
Robert Haas wrote: I'm wondering why md.c is converting the results from an exact value to a floating point, only to have xlog.c turn around and convert back to an integer. I think it could just return milliseconds directly, or if you're worried about a checkpoint that takes more than 24 days

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-13 Thread Robert Haas
On Mon, Dec 13, 2010 at 3:19 AM, Greg Smith g...@2ndquadrant.com wrote: Robert Haas wrote: I'm wondering why md.c is converting the results from an exact value to a floating point, only to have xlog.c turn around and convert back to an integer.  I think it could just return milliseconds

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-12 Thread Robert Haas
On Sun, Dec 5, 2010 at 4:23 PM, Greg Smith g...@2ndquadrant.com wrote: Jeff Janes wrote: I've attached a tiny patch to apply over yours, to deal with this and with the case where no files are synced. Thanks for that.  That obvious error eluded me because in most of the patch update testing

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-09 Thread Greg Smith
Jeff Janes wrote: In my test cases, the syncs that the backends were doing were almost always to the same file that the checkpoint writer was already choking on (so they are entangled simply by virtue of that). So very quickly all the backends hit the same wall and thunked to a halt. This is

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-07 Thread Jeff Janes
On Sun, Dec 5, 2010 at 1:23 PM, Greg Smith g...@2ndquadrant.com wrote: Jeff Janes wrote: I've attached a tiny patch to apply over yours, to deal with this and with the case where no files are synced. Thanks for that.  That obvious error eluded me because in most of the patch update testing

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-05 Thread Greg Smith
Jeff Janes wrote: I've attached a tiny patch to apply over yours, to deal with this and with the case where no files are synced. Thanks for that. That obvious error eluded me because in most of the patch update testing I was doing (on ext3), the longest sync was always about the same

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-02 Thread Jeff Janes
On Tue, Nov 30, 2010 at 12:15 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Tue, Nov 30, 2010 at 8:38 AM, Greg Smith g...@2ndquadrant.com wrote: Hi Greg, Thanks for the update. This might be ready for some proper review now.  I know there's at least one blatant bug still in here I

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-30 Thread Greg Smith
Jeff Janes wrote: For the individual file sync times emitted under debug1, it would be very handy if the file being synced was identified, for example relation base/16384/16523. Rather than being numbered sequentially within a given checkpoint. I was numbering them sequentially so that it's

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-30 Thread Cédric Villemain
2010/11/30 Greg Smith g...@2ndquadrant.com: Jeff Janes wrote: For the individual file sync times emitted under debug1, it would be very handy if the file being synced was identified, for example relation base/16384/16523. Rather than being numbered sequentially within a given checkpoint.

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-30 Thread Jeff Janes
On Tue, Nov 30, 2010 at 8:38 AM, Greg Smith g...@2ndquadrant.com wrote: Hi Greg, Thanks for the update. This might be ready for some proper review now.  I know there's at least one blatant bug still in here I haven't found yet, related to how the averages are computed. Yes, the blatant

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-26 Thread Greg Smith
Jeff Janes wrote: For the individual file sync times emitted under debug1, it would be very handy if the file being synced was identified, for example relation base/16384/16523. I was in the middle of looking into adding that already, so consider that part of the target for the next update

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-24 Thread Robert Haas
On Wed, Nov 24, 2010 at 1:02 AM, Greg Smith g...@2ndquadrant.com wrote: Robert Haas wrote: Did this get eaten by the email goblin, or you're still working on it? Fell behind due to an unfortunately timed bit of pneumonia.  Hurray for the health benefits of cross country flights.  Will fix

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-23 Thread Greg Smith
Robert Haas wrote: Did this get eaten by the email goblin, or you're still working on it? Fell behind due to an unfortunately timed bit of pneumonia. Hurray for the health benefits of cross country flights. Will fix this up, rebase my other patch, and head toward some more review/'Fest

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-22 Thread Jeff Janes
On Mon, Nov 15, 2010 at 12:09 PM, Greg Smith g...@2ndquadrant.com wrote: So my task list is: 0) Rebase against the HEAD that just code related to this touched today 1) Assume that log_checkpoints is sufficient control over whether the timing overhead added is worth collecting, and therefore

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-21 Thread Robert Haas
On Mon, Nov 15, 2010 at 3:09 PM, Greg Smith g...@2ndquadrant.com wrote: So my task list is: 0) Rebase against the HEAD that just code related to this touched today 1) Assume that log_checkpoints is sufficient control over whether the timing overhead added is worth collecting, and therefore

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-15 Thread Robert Haas
On Sun, Nov 14, 2010 at 7:04 PM, Greg Smith g...@2ndquadrant.com wrote: It might.  One trade-off is that if you're looking at the sync write detail, the summary comes out in a similar form.  And it was easy to put in here--I'd have to return some new data out of the sync phase call in order

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-15 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I would be very surprised if we can find a system where gettimeofday() takes a significant amount of time compared with fsync(). It might be (probably is) too expensive to stick into code paths that are heavily CPU-bounded, but surely the cost here is

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-15 Thread Greg Smith
So my task list is: 0) Rebase against the HEAD that just code related to this touched today 1) Assume that log_checkpoints is sufficient control over whether the timing overhead added is worth collecting, and therefore remove the half-baked idea of also wrapping with a compile-time option.

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-15 Thread Robert Haas
On Mon, Nov 15, 2010 at 2:48 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I would be very surprised if we can find a system where gettimeofday() takes a significant amount of time compared with fsync().  It might be (probably is) too expensive to stick into

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-14 Thread Magnus Hagander
On Sun, Nov 14, 2010 at 22:37, Greg Smith g...@2ndquadrant.com wrote: Attached patch adds some logging for each individual fsync call made during a checkpoint, along with a summary at the end.  You need to have the following to see all of the detail: log_checkpoints=on

Re: [HACKERS] Instrument checkpoint sync calls

2010-11-14 Thread Greg Smith
Magnus Hagander wrote: I think that it's reasonable for the sort of people who turn log_checkpoints on to also get the sync summary line, thus it being logged at LOG level. In that case, wouldn't it make more sense to add a couple of more fields to the existing line? Easier to