Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-05-02 Thread Tom Lane
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > What we could have is the semantics of "Return a buffer, with either > correct contents or completely zeroed out". It would act just like > ReadBuffer if the buffer was already in memory, and zero out the page > otherwise. That's a bit strange sem

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-28 Thread Tom Lane
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > I was actually thinking that we could slip this in 8.3. It's a simple, > well-understood patch, which fixes a little data integrity quirk as well > as gives a nice recovery speed up. Yeah. It's arguably a bug fix, in fact, since it eliminates the

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-28 Thread Heikki Linnakangas
I was actually thinking that we could slip this in 8.3. It's a simple, well-understood patch, which fixes a little data integrity quirk as well as gives a nice recovery speed up. Bruce Momjian wrote: I assume this is 8.4 material. --

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2007-04-27 at 12:22 +0100, Heikki Linnakangas wrote: Patch implementing that attached. I named the function "ReadOrZeroBuffer". We already have an API quirk similar to this: relation extension. It seems strange to have two different kinds of special case API that are

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-28 Thread Simon Riggs
On Fri, 2007-04-27 at 12:22 +0100, Heikki Linnakangas wrote: > Tom Lane wrote: > > "Simon Riggs" <[EMAIL PROTECTED]> writes: > >> As regards the zero_damaged_pages question, I raised that some time ago > >> but we didn't arrive at an explicit answer. All I would say is we can't > >> allow invalid p

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Jim Nasby
On Apr 26, 2007, at 3:39 PM, Tom Lane wrote: Jim Nasby <[EMAIL PROTECTED]> writes: So what happens if a backend is running with full_page_writes = off, someone edits postgresql.conf to turns it on and forgets to reload/ restart, and then we crash? You'll come up in recovery mode thinking that f_

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Simon Riggs
On Fri, 2007-04-27 at 10:37 -0400, Bruce Momjian wrote: > I assume this is 8.4 material. I think its a small enough, performance-only change to allow it at this time. It will provide considerable additional benefit for Warm Standby servers. -- Simon Riggs EnterpriseDB http://w

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Bruce Momjian
I assume this is 8.4 material. --- Heikki Linnakangas wrote: > Tom Lane wrote: > > "Simon Riggs" <[EMAIL PROTECTED]> writes: > >> As regards the zero_damaged_pages question, I raised that some time ago > >> but we didn't arr

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Alvaro Herrera
Heikki Linnakangas wrote: > Alvaro Herrera wrote: > >Heikki Linnakangas wrote: > > > >>What we could have is the semantics of "Return a buffer, with either > >>correct contents or completely zeroed out". It would act just like > >>ReadBuffer if the buffer was already in memory, and zero out the p

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Heikki Linnakangas
Alvaro Herrera wrote: Heikki Linnakangas wrote: What we could have is the semantics of "Return a buffer, with either correct contents or completely zeroed out". It would act just like ReadBuffer if the buffer was already in memory, and zero out the page otherwise. That's a bit strange semanti

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Alvaro Herrera
Heikki Linnakangas wrote: > What we could have is the semantics of "Return a buffer, with either > correct contents or completely zeroed out". It would act just like > ReadBuffer if the buffer was already in memory, and zero out the page > otherwise. That's a bit strange semantics to have, but

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-27 Thread Heikki Linnakangas
Tom Lane wrote: "Simon Riggs" <[EMAIL PROTECTED]> writes: As regards the zero_damaged_pages question, I raised that some time ago but we didn't arrive at an explicit answer. All I would say is we can't allow invalid pages in the buffer manager at any time, whatever options we have requested, oth

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-26 Thread Tom Lane
Jim Nasby <[EMAIL PROTECTED]> writes: > So what happens if a backend is running with full_page_writes = off, > someone edits postgresql.conf to turns it on and forgets to reload/ > restart, and then we crash? You'll come up in recovery mode thinking > that f_p_w was turned on, when in fact it

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-26 Thread Zeugswetter Andreas ADI SD
> So what happens if a backend is running with full_page_writes > = off, someone edits postgresql.conf to turns it on and > forgets to reload/ restart, and then we crash? You'll come up > in recovery mode thinking that f_p_w was turned on, when in > fact it wasn't. > > ISTM that we need to so

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-26 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas <[EMAIL PROTECTED]> writes: Tom Lane wrote: but it'd make it safe to use in non-WAL contexts (I think there are other places where we know we are going to init the page and so a physical read is a waste of time). Is there? I can't think of any. Extending a

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-26 Thread Jim Nasby
On Apr 25, 2007, at 2:48 PM, Heikki Linnakangas wrote: In recovery, with full_pages_writes=on, we read in each page only to overwrite the contents with a full page image. That's a waste of time, and can have a surprisingly large effect on recovery time. As a quick test on my laptop, I initia

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Tom Lane
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> but it'd make it safe to use in non-WAL contexts (I think there are >> other places where we know we are going to init the page and so a >> physical read is a waste of time). > Is there? I can't think of any. Extending a relation

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Heikki Linnakangas
Tom Lane wrote: "Simon Riggs" <[EMAIL PROTECTED]> writes: As regards the zero_damaged_pages question, I raised that some time ago but we didn't arrive at an explicit answer. All I would say is we can't allow invalid pages in the buffer manager at any time, whatever options we have requested, oth

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Tom Lane
"Simon Riggs" <[EMAIL PROTECTED]> writes: > As regards the zero_damaged_pages question, I raised that some time ago > but we didn't arrive at an explicit answer. All I would say is we can't > allow invalid pages in the buffer manager at any time, whatever options > we have requested, otherwise othe

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Simon Riggs
On Wed, 2007-04-25 at 13:48 +0100, Heikki Linnakangas wrote: > I was surprised how big a difference it makes, but when you think about > it it's logical. Without the patch, it's doing roughly the same I/O as > the test itself, reading in pages, modifying them, and writing them > back. With the

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Heikki Linnakangas
Gregory Stark wrote: So in short I think with your patch this piece of code no longer has a role. Either your patch kicks in and we never even look at the damaged page at all, or we should be treating it as corrupt data and just check zero_damaged_pages alone and not do anything special in recove

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Gregory Stark
"Heikki Linnakangas" <[EMAIL PROTECTED]> writes: > While working on this, this comment in ReadBuffer caught my eye: > >> /* >> * During WAL recovery, the first access to any data page should >> * overwrite the whole page from the WAL; so a clobbered page >> * header is not r

Re: [HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Heikki Linnakangas
Heikki Linnakangas wrote: While working on this, this comment in ReadBuffer caught my eye: /* * During WAL recovery, the first access to any data page should * overwrite the whole page from the WAL; so a clobbered page * header is not reason to fail. Hence, when InRecovery w

[HACKERS] Avoiding unnecessary reads in recovery

2007-04-25 Thread Heikki Linnakangas
In recovery, with full_pages_writes=on, we read in each page only to overwrite the contents with a full page image. That's a waste of time, and can have a surprisingly large effect on recovery time. As a quick test on my laptop, I initialized a DBT-2 test with 5 warehouses, and let it run for