Re: [HACKERS] CLOG extension

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 9:11 AM, Simon Riggs wrote: > On 4 May 2012 13:59, Robert Haas wrote: >> On Fri, May 4, 2012 at 3:35 AM, Simon Riggs wrote: >>> On Thu, May 3, 2012 at 9:56 PM, Robert Haas wrote: On Thu, May 3, 2012 at 3:20 PM, Simon Riggs wrote: > Your two paragraphs have rough

Re: [HACKERS] CLOG extension

2012-05-04 Thread Simon Riggs
On 4 May 2012 13:59, Robert Haas wrote: > On Fri, May 4, 2012 at 3:35 AM, Simon Riggs wrote: >> On Thu, May 3, 2012 at 9:56 PM, Robert Haas wrote: >>> On Thu, May 3, 2012 at 3:20 PM, Simon Riggs wrote: Your two paragraphs have roughly opposite arguments... Doing it every 32 pages

Re: [HACKERS] CLOG extension

2012-05-04 Thread Robert Haas
On Fri, May 4, 2012 at 3:35 AM, Simon Riggs wrote: > On Thu, May 3, 2012 at 9:56 PM, Robert Haas wrote: >> On Thu, May 3, 2012 at 3:20 PM, Simon Riggs wrote: >>> Your two paragraphs have roughly opposite arguments... >>> >>> Doing it every 32 pages would give you 30 seconds to complete the >>> f

Re: [HACKERS] CLOG extension

2012-05-04 Thread Simon Riggs
On Thu, May 3, 2012 at 9:56 PM, Robert Haas wrote: > On Thu, May 3, 2012 at 3:20 PM, Simon Riggs wrote: >> Your two paragraphs have roughly opposite arguments... >> >> Doing it every 32 pages would give you 30 seconds to complete the >> fsync, if you kicked it off when half way through the previo

Re: [HACKERS] CLOG extension

2012-05-03 Thread Daniel Farina
On Thu, May 3, 2012 at 2:34 PM, Tom Lane wrote: > Alvaro Herrera writes: >> Excerpts from Daniel Farina's message of jue may 03 17:04:03 -0400 2012: >>> I sort of care about this, but only on systems that are not very busy >>> and could otherwise get by with fewer resources -- for example, it'd >

Re: [HACKERS] CLOG extension

2012-05-03 Thread Tom Lane
Alvaro Herrera writes: > Excerpts from Daniel Farina's message of jue may 03 17:04:03 -0400 2012: >> I sort of care about this, but only on systems that are not very busy >> and could otherwise get by with fewer resources -- for example, it'd >> be nice to turn off autovacuum and the stat collecto

Re: [HACKERS] CLOG extension

2012-05-03 Thread Daniel Farina
On Thu, May 3, 2012 at 2:26 PM, Alvaro Herrera wrote: > I'm not sure I see the point in worrying about this at all.  I mean, a > process doing nothing does not waste much resources, does it?  Other > than keeping a PID that you can't use for other stuff. Not much, but we do have an interest in ve

Re: [HACKERS] CLOG extension

2012-05-03 Thread Alvaro Herrera
Excerpts from Daniel Farina's message of jue may 03 17:04:03 -0400 2012: > On Thu, May 3, 2012 at 1:56 PM, Robert Haas wrote: > > Possibly.  I have some fear of ending up with too many background > > processes, but we may need them. > > I sort of care about this, but only on systems that are not

Re: [HACKERS] CLOG extension

2012-05-03 Thread Daniel Farina
On Thu, May 3, 2012 at 1:56 PM, Robert Haas wrote: > Possibly.  I have some fear of ending up with too many background > processes, but we may need them. I sort of care about this, but only on systems that are not very busy and could otherwise get by with fewer resources -- for example, it'd be n

Re: [HACKERS] CLOG extension

2012-05-03 Thread Robert Haas
On Thu, May 3, 2012 at 3:20 PM, Simon Riggs wrote: > Your two paragraphs have roughly opposite arguments... > > Doing it every 32 pages would give you 30 seconds to complete the > fsync, if you kicked it off when half way through the previous file - > at current maximum rates. So there is utility

Re: [HACKERS] CLOG extension

2012-05-03 Thread Simon Riggs
On Thu, May 3, 2012 at 7:50 PM, Robert Haas wrote: > On Thu, May 3, 2012 at 1:27 PM, Simon Riggs wrote: >> Why not switch to 1 WAL record per file, rather than 1 per page. (32 >> pages, IIRC). >> >> We can then have the whole new file written as zeroes by a background >> process, which needn't do

Re: [HACKERS] CLOG extension

2012-05-03 Thread Robert Haas
On Thu, May 3, 2012 at 2:50 PM, Robert Haas wrote: > Doing it a background process, though, may make sense.  What I'm a > little worried about is that - on a busy system - we've only got about > 2 seconds to complete each CLOG extension, and we must do an fsync in > order to get there. Scratch th

Re: [HACKERS] CLOG extension

2012-05-03 Thread Robert Haas
On Thu, May 3, 2012 at 1:27 PM, Simon Riggs wrote: > Why not switch to 1 WAL record per file, rather than 1 per page. (32 > pages, IIRC). > > We can then have the whole new file written as zeroes by a background > process, which needn't do that while holding the XidGenLock. I thought about doing

Re: [HACKERS] CLOG extension

2012-05-03 Thread Simon Riggs
On Thu, May 3, 2012 at 5:59 PM, Tom Lane wrote: > Robert Haas writes: >> [ CLOG extension is horrid for concurrency ] > > Yeah.  When that code was designed, a page's worth of transactions > seemed like a lot so we didn't worry too much about performance glitches > when we crossed a page boundary

Re: [HACKERS] CLOG extension

2012-05-03 Thread Tom Lane
Robert Haas writes: > [ CLOG extension is horrid for concurrency ] Yeah. When that code was designed, a page's worth of transactions seemed like a lot so we didn't worry too much about performance glitches when we crossed a page boundary. It's time to do something about it though. The idea of

[HACKERS] CLOG extension

2012-05-03 Thread Robert Haas
Currently, the following can happen: 1. A backend needs a new transaction, so it calls GetNewTransactionId(). It acquires XidGenLock and then calls ExtendCLOG(). 2. ExtendCLOG() decides that a new CLOG page is needed, so it acquires CLogControlLock and then calls ZeroCLOGPage(). 3. ZeroCLOGPage()