Re: [HACKERS] Separating bgwriter and checkpointer

2011-11-01 Thread Simon Riggs
On Wed, Oct 19, 2011 at 3:58 PM, Simon Riggs wrote: > On Wed, Oct 19, 2011 at 3:29 PM, Fujii Masao wrote: >> On Wed, Oct 19, 2011 at 9:45 PM, Robert Haas wrote: >>> I don't really see any reason to break the monitoring view just >>> because we did some internal refactoring.  I'd rather have back

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-24 Thread Simon Riggs
On Mon, Oct 24, 2011 at 11:40 AM, Heikki Linnakangas wrote: > The patch looks sane, it's mostly just moving existing code around, but > there's one thing that's been bothering me about this whole idea from the > get-go: > > If the bgwriter and checkpointer are two different processes, whenever >

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-24 Thread Heikki Linnakangas
On 19.10.2011 17:58, Simon Riggs wrote: On Wed, Oct 19, 2011 at 3:29 PM, Fujii Masao wrote: On Wed, Oct 19, 2011 at 9:45 PM, Robert Haas wrote: I don't really see any reason to break the monitoring view just because we did some internal refactoring. I'd rather have backward compatibility.

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-19 Thread Simon Riggs
On Wed, Oct 19, 2011 at 3:29 PM, Fujii Masao wrote: > On Wed, Oct 19, 2011 at 9:45 PM, Robert Haas wrote: >> I don't really see any reason to break the monitoring view just >> because we did some internal refactoring.  I'd rather have backward >> compatibility. > > Fair enough. > > The patch does

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-19 Thread Fujii Masao
On Wed, Oct 19, 2011 at 9:45 PM, Robert Haas wrote: > I don't really see any reason to break the monitoring view just > because we did some internal refactoring.  I'd rather have backward > compatibility. Fair enough. The patch doesn't change any document, but at least the description of pg_stat

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-19 Thread Robert Haas
On Wed, Oct 19, 2011 at 8:43 AM, Dickson S. Guedes wrote: > 2011/10/19 Fujii Masao : >> On Tue, Oct 18, 2011 at 10:18 PM, Simon Riggs wrote: >>> Any reason or objection to committing this patch? >> >> The checkpointer doesn't call pgstat_send_bgwriter(), but it should. >> Otherwise, some entries

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-19 Thread Dickson S. Guedes
2011/10/19 Fujii Masao : > On Tue, Oct 18, 2011 at 10:18 PM, Simon Riggs wrote: >> Any reason or objection to committing this patch? > > The checkpointer doesn't call pgstat_send_bgwriter(), but it should. > Otherwise, some entries in pg_stat_bgwriter will never be updated. Yes, checkpoints_req,

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-19 Thread Dickson S. Guedes
2011/10/18 Simon Riggs : > On Wed, Oct 5, 2011 at 8:02 AM, Simon Riggs wrote: >> On Wed, Oct 5, 2011 at 5:10 AM, Dickson S. Guedes >> wrote: >> >>> Ah ok! I started reviewing the v4 patch version, this is my comments: >> >> ... >> >>> Well, all the tests was running with the default postgresql.c

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-19 Thread Fujii Masao
On Tue, Oct 18, 2011 at 10:18 PM, Simon Riggs wrote: > Any reason or objection to committing this patch? The checkpointer doesn't call pgstat_send_bgwriter(), but it should. Otherwise, some entries in pg_stat_bgwriter will never be updated. If we adopt the patch, checkpoint is performed by check

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-18 Thread Robert Haas
On Tue, Oct 18, 2011 at 12:53 PM, Simon Riggs wrote: > On Tue, Oct 18, 2011 at 5:39 PM, Robert Haas wrote: >> On Tue, Oct 18, 2011 at 9:18 AM, Simon Riggs wrote: >>> Any reason or objection to committing this patch? >> >> Not on my end, though I haven't reviewed it in detail.  One minor note >>

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-18 Thread Simon Riggs
On Tue, Oct 18, 2011 at 5:39 PM, Robert Haas wrote: > On Tue, Oct 18, 2011 at 9:18 AM, Simon Riggs wrote: >> Any reason or objection to committing this patch? > > Not on my end, though I haven't reviewed it in detail.  One minor note > - I was mildly surprised to see that you moved this to the >

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-18 Thread Robert Haas
On Tue, Oct 18, 2011 at 9:18 AM, Simon Riggs wrote: > Any reason or objection to committing this patch? Not on my end, though I haven't reviewed it in detail. One minor note - I was mildly surprised to see that you moved this to the checkpointer rather than leaving it in the bgwriter: + /

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-18 Thread Simon Riggs
On Wed, Oct 5, 2011 at 8:02 AM, Simon Riggs wrote: > On Wed, Oct 5, 2011 at 5:10 AM, Dickson S. Guedes > wrote: > >> Ah ok! I started reviewing the v4 patch version, this is my comments: > > ... > >> Well, all the tests was running with the default postgresql.conf in my >> laptop but I'll setup

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-05 Thread Simon Riggs
On Wed, Oct 5, 2011 at 5:10 AM, Dickson S. Guedes wrote: > Ah ok! I started reviewing the v4 patch version, this is my comments: ... > Well, all the tests was running with the default postgresql.conf in my > laptop but I'll setup a more "real world" environment to test for > performance regress

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-04 Thread Dickson S. Guedes
2011/10/4 Simon Riggs : > The problem is the *same* one I fixed in v2, yet now I see I managed > to somehow exclude that fix from the earlier patch. Slap. Anyway, > fixed again now. Ah ok! I started reviewing the v4 patch version, this is my comments: Submission review === 1. The pat

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-04 Thread Simon Riggs
On Tue, Oct 4, 2011 at 2:51 AM, Dickson S. Guedes wrote: > 2011/10/3 Simon Riggs : >> On Sun, Oct 2, 2011 at 11:45 PM, Dickson S. Guedes >> wrote: >>> I'm trying your patch, it was applied cleanly to master and compiled >>> ok. But since I started postgres I'm seeing a  99% of CPU usage: >> >> O

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-03 Thread Dickson S. Guedes
2011/10/3 Simon Riggs : > On Sun, Oct 2, 2011 at 11:45 PM, Dickson S. Guedes > wrote: >> I'm trying your patch, it was applied cleanly to master and compiled >> ok. But since I started postgres I'm seeing a  99% of CPU usage: > > Oh, thanks. I see what happened. I was toying with the idea of goin

Re: [HACKERS] Separating bgwriter and checkpointer

2011-10-02 Thread Dickson S. Guedes
2011/10/2 Simon Riggs : > On Thu, Sep 15, 2011 at 11:53 PM, Simon Riggs wrote: > >> Current patch has a bug at shutdown I've not located yet, but seems >> likely is a simple error. That is mainly because for personal reasons >> I've not been able to work on the patch recently. I expect to be able

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-21 Thread Greg Smith
On 09/20/2011 09:35 AM, Heikki Linnakangas wrote: Yeah, I was thinking of write()s, not fsyncs. I agree this might have some effect during fsync phase. Right; that's where the most serious problems seem to pop up at anyway now. All the testing I did earlier this year suggested Linux at least

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Robert Haas
On Tue, Sep 20, 2011 at 11:01 AM, Marti Raudsepp wrote: > On Fri, Sep 16, 2011 at 01:53, Simon Riggs wrote: >> This patch splits bgwriter into 2 processes: checkpointer and >> bgwriter, seeking to avoid contentious changes. Additional changes are >> expected in this release to build upon these ch

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Marti Raudsepp
On Fri, Sep 16, 2011 at 01:53, Simon Riggs wrote: > This patch splits bgwriter into 2 processes: checkpointer and > bgwriter, seeking to avoid contentious changes. Additional changes are > expected in this release to build upon these changes for both new > processes, though this patch stands on it

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Robert Haas
On Tue, Sep 20, 2011 at 9:35 AM, Heikki Linnakangas wrote: >> All that said my question is which way is the code more legible and >> easier to follow? > > Hear hear. If we're going to give the bgwriter more responsibilities, this > might make sense even if it has no effect on performance. I agree

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Heikki Linnakangas
On 20.09.2011 17:31, Cédric Villemain wrote: 2011/9/20 Heikki Linnakangas: On 20.09.2011 16:49, Magnus Hagander wrote: Isn't there also the advantage of that work put in two different processes can use two different CPU cores? Or is that likely to never ever come in play here? You would need

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Cédric Villemain
2011/9/20 Heikki Linnakangas : > On 20.09.2011 16:49, Magnus Hagander wrote: >> >> Isn't there also the advantage of that work put in two different >> processes can use two different CPU cores? Or is that likely to never >> ever come in play here? > > You would need one helluva I/O system to satura

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Heikki Linnakangas
On 20.09.2011 16:49, Magnus Hagander wrote: Isn't there also the advantage of that work put in two different processes can use two different CPU cores? Or is that likely to never ever come in play here? You would need one helluva I/O system to saturate even a single CPU, just by doing write+fs

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Magnus Hagander
On Tue, Sep 20, 2011 at 15:35, Heikki Linnakangas wrote: > On 20.09.2011 16:29, Greg Stark wrote: >> >> On Tue, Sep 20, 2011 at 11:03 AM, Simon Riggs >>  wrote: I don't see what difference it makes which process does the I/O. If a write() by checkpointer process blocks, any write()s

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Heikki Linnakangas
On 20.09.2011 16:29, Greg Stark wrote: On Tue, Sep 20, 2011 at 11:03 AM, Simon Riggs wrote: I don't see what difference it makes which process does the I/O. If a write() by checkpointer process blocks, any write()s by the separate bgwriter process at that time will block too. If the I/O is not

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Greg Stark
On Tue, Sep 20, 2011 at 11:03 AM, Simon Riggs wrote: >> I don't see what difference it makes which process does the I/O. If a >> write() by checkpointer process blocks, any write()s by the separate >> bgwriter process at that time will block too. If the I/O is not saturated, >> and the checkpoint

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Simon Riggs
On Tue, Sep 20, 2011 at 10:03 AM, Heikki Linnakangas wrote: > On 20.09.2011 11:18, Simon Riggs wrote: >> >> The bgwriter avoids I/O, if it is operating correctly. This patch >> ensures it continues to operate even during heavy checkpoints. So it >> helps avoid extra I/O during a period of very hig

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Heikki Linnakangas
On 20.09.2011 11:18, Simon Riggs wrote: The bgwriter avoids I/O, if it is operating correctly. This patch ensures it continues to operate even during heavy checkpoints. So it helps avoid extra I/O during a period of very high I/O activity. I don't see what difference it makes which process does

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Simon Riggs
On Tue, Sep 20, 2011 at 9:06 AM, Heikki Linnakangas wrote: > On 20.09.2011 10:48, Simon Riggs wrote: >> >> On Fri, Sep 16, 2011 at 2:38 AM, Fujii Masao >>  wrote: >>> >>> On Fri, Sep 16, 2011 at 7:53 AM, Simon Riggs >>>  wrote: This patch splits bgwriter into 2 processes: checkpointer an

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Heikki Linnakangas
On 20.09.2011 10:48, Simon Riggs wrote: On Fri, Sep 16, 2011 at 2:38 AM, Fujii Masao wrote: On Fri, Sep 16, 2011 at 7:53 AM, Simon Riggs wrote: This patch splits bgwriter into 2 processes: checkpointer and bgwriter, seeking to avoid contentious changes. Additional changes are expected in this

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-20 Thread Simon Riggs
On Fri, Sep 16, 2011 at 2:38 AM, Fujii Masao wrote: > On Fri, Sep 16, 2011 at 7:53 AM, Simon Riggs wrote: >> This patch splits bgwriter into 2 processes: checkpointer and >> bgwriter, seeking to avoid contentious changes. Additional changes are >> expected in this release to build upon these chan

Re: [HACKERS] Separating bgwriter and checkpointer

2011-09-15 Thread Fujii Masao
On Fri, Sep 16, 2011 at 7:53 AM, Simon Riggs wrote: > This patch splits bgwriter into 2 processes: checkpointer and > bgwriter, seeking to avoid contentious changes. Additional changes are > expected in this release to build upon these changes for both new > processes, though this patch stands on