Re: [HACKERS] switch UNLOGGED to LOGGED

2011-06-03 Thread Robert Haas
On Tue, May 31, 2011 at 12:25 PM, Leonardo Francalanci m_li...@yahoo.it wrote: Well, I  sort of assumed the design was OK, too, but the more we talk about this  WAL-logging stuff, the less convinced I am that I really understand the  problem.  :-( I see. In fact, I think nobody thought about

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-31 Thread Leonardo Francalanci
I think we need a detailed design document for how this is all going to work. We need to not only handle the master properly but also handle the slave properly. Consider, for example, the case where the slave begins to replay the transaction, reaches a restartpoint after replaying some

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-31 Thread Robert Haas
On Tue, May 31, 2011 at 3:39 AM, Leonardo Francalanci m_li...@yahoo.it wrote: I think we need a detailed design document for how  this is all going to work. We need to not only handle the master properly but  also handle the slave properly.  Consider, for example, the case where  the slave

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-31 Thread Leonardo Francalanci
Well, I sort of assumed the design was OK, too, but the more we talk about this WAL-logging stuff, the less convinced I am that I really understand the problem. :-( I see. In fact, I think nobody thought about restart points... To sum up: 1) everything seems ok when in the wal_level =

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-30 Thread Leonardo Francalanci
Why is it necessary to replay the operation only on the slave? Can we just use XLOG_HEAP_NEWPAGE? Uh, I don't know why but I thought I shouldn't log a page on the master, since all the pages are already there and fsync-ed. But if it makes no harm, I can easily use XLOG_HEAP_NEWPAGE (of

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-30 Thread Robert Haas
On Sun, May 29, 2011 at 4:29 AM, Noah Misch n...@leadboat.com wrote: I don't think it is *necessary*.  If we're replaying WAL on a master, we'll also be resetting unlogged relations after recovery; what we write or do not write to them in the mean time has no functional impact.  Seemed like

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-29 Thread Noah Misch
On Sat, May 28, 2011 at 09:33:09PM -0400, Robert Haas wrote: On Fri, May 27, 2011 at 6:19 AM, Noah Misch n...@leadboat.com wrote: So, it's ok to have a log item that is replayed only if WalRcvInProgress() is true? No, that checks for WAL streaming in particular. ?A log-shipping

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-28 Thread Robert Haas
On Fri, May 27, 2011 at 6:19 AM, Noah Misch n...@leadboat.com wrote: So, it's ok to have a log item that is replayed only if WalRcvInProgress() is true? No, that checks for WAL streaming in particular.  A log-shipping standby needs the same treatment. Is it a correct approach? I couldn't

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-27 Thread Noah Misch
On Fri, May 20, 2011 at 09:37:20AM +0100, Leonardo Francalanci wrote: I'll try to sum up what I understood: 1) the standby keeps the lock, so no problem with stray files coming from the unlogged-logged log reply, as the table can't be read during the operation 2) calling

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-27 Thread Leonardo Francalanci
From: Noah Misch n...@leadboat.com - the patch is missing the send all table pages to the standby part; is there some code I can use as base? Nothing comes to mind as especially similar. I guess I have to generate some special log type that is only played by standby servers.

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-27 Thread Noah Misch
On Fri, May 27, 2011 at 10:49:13AM +0100, Leonardo Francalanci wrote: From: Noah Misch n...@leadboat.com - the patch is missing the send all table pages to the standby part; is there some code I can use as base? Nothing comes to mind as especially similar. I guess I have to

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-20 Thread Leonardo Francalanci
I'll try to sum up what I understood: 1) the standby keeps the lock, so no problem with stray files coming from the unlogged-logged log reply, as the table can't be read during the operation 2) calling ResetUnloggedRelations before ProcArrayApplyRecoveryInfo would remove the problem of the

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-20 Thread Leonardo Francalanci
- the patch is missing the send all table pages to the standby part; is there some code I can use as base? I guess I have to generate some special log type that is only played by standby servers. Maybe I could use log_newpage, but instead of XLOG_HEAP_NEWPAGE I could use something like

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-19 Thread Leonardo Francalanci
On Wed, May 18, 2011 at 04:02:59PM +0100, Leonardo Francalanci wrote: By the time the startup process releases the AccessExclusiveLock acquired by the proposed UNLOGGED - normal conversion process, that relfilenode needs to be either fully copied or unlinked all over again.

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-19 Thread Noah Misch
On Thu, May 19, 2011 at 01:52:46PM +0100, Leonardo Francalanci wrote: I'd guess some WAL record arising from the post-crash master restart makes the standby do so. When a crash isn't involved, the commit or abort record is that signal. You could test and find out how it happens

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-19 Thread Robert Haas
On Thu, May 19, 2011 at 11:13 AM, Noah Misch n...@leadboat.com wrote: It would solve the problem, but it would mean resetting unlogged relations on the standby at every shutdown checkpoint.  That's probably not a performance problem, but it is a hack. I haven't thought about this too deeply,

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-19 Thread Noah Misch
On Thu, May 19, 2011 at 11:42:03AM -0400, Robert Haas wrote: On Thu, May 19, 2011 at 11:13 AM, Noah Misch n...@leadboat.com wrote: It would solve the problem, but it would mean resetting unlogged relations on the standby at every shutdown checkpoint. ?That's probably not a performance

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-19 Thread Robert Haas
On Thu, May 19, 2011 at 3:24 PM, Noah Misch n...@leadboat.com wrote: On Thu, May 19, 2011 at 11:42:03AM -0400, Robert Haas wrote: On Thu, May 19, 2011 at 11:13 AM, Noah Misch n...@leadboat.com wrote: It would solve the problem, but it would mean resetting unlogged relations on the standby

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-19 Thread Noah Misch
On Thu, May 19, 2011 at 03:53:12PM -0400, Robert Haas wrote: On Thu, May 19, 2011 at 3:24 PM, Noah Misch n...@leadboat.com wrote: On Thu, May 19, 2011 at 11:42:03AM -0400, Robert Haas wrote: On Thu, May 19, 2011 at 11:13 AM, Noah Misch n...@leadboat.com wrote: It would solve the problem,

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-18 Thread Leonardo Francalanci
By the time the startup process releases the AccessExclusiveLock acquired by the proposed UNLOGGED - normal conversion process, that relfilenode needs to be either fully copied or unlinked all over again. (Alternately, find some other way to make sure queries don't read the half-copied

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-18 Thread Noah Misch
On Wed, May 18, 2011 at 04:02:59PM +0100, Leonardo Francalanci wrote: By the time the startup process releases the AccessExclusiveLock acquired by the proposed UNLOGGED - normal conversion process, that relfilenode needs to be either fully copied or unlinked all over again.

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-10 Thread Leonardo Francalanci
Yes, that seems like a very appealing approach. There is plenty of bit-space available in xinfo, and we could reserve a bit each for nrels, nsubxacts, and nmsgs, with set meaning that an integer count of that item is present and clear meaning that the count is omitted from the structure

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-10 Thread Robert Haas
On Tue, May 10, 2011 at 3:35 AM, Leonardo Francalanci m_li...@yahoo.it wrote: Yes, that seems like a very appealing approach.   There is plenty of bit-space available in xinfo, and we could reserve a bit  each for nrels, nsubxacts, and nmsgs, with set meaning that an integer count  of that

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-10 Thread Leonardo Francalanci
I don't think making xinfo shorter will save anything, because whatever follows it is going to be a 4-byte quantity and therefore 4-byte aligned. ups, didn't notice it. I'll splitxinfo into: uint16 xinfo; uint16 presentFlags; I guess it helps with the reading? I mean, instead

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-10 Thread Robert Haas
On Tue, May 10, 2011 at 8:03 AM, Leonardo Francalanci m_li...@yahoo.it wrote: I don't  think making xinfo shorter will save anything, because whatever follows it is  going to be a 4-byte quantity and therefore 4-byte aligned. ups, didn't notice it. I'll split    xinfo into: uint16  

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-09 Thread Alvaro Herrera
Excerpts from Robert Haas's message of vie may 06 23:25:09 -0300 2011: On Fri, Apr 22, 2011 at 4:13 AM, Leonardo Francalanci m_li...@yahoo.it wrote: Maybe you should change  xl_act_commit to have a separate list of rels to drop the init fork for  (instead of mixing those with the list of

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-09 Thread Robert Haas
On Mon, May 9, 2011 at 12:51 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of vie may 06 23:25:09 -0300 2011: On Fri, Apr 22, 2011 at 4:13 AM, Leonardo Francalanci m_li...@yahoo.it wrote: Maybe you should change  xl_act_commit to have a separate

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-08 Thread Leonardo Francalanci
On Fri, Apr 22, 2011 at 4:13 AM, Leonardo Francalanci m_li...@yahoo.it wrote: Maybe you should change xl_act_commit to have a separate list of rels to drop the init fork for (instead of mixing those with the list of files to drop as a whole). I tried to follow your suggestion,

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, May 6, 2011 at 10:25 PM, Robert Haas robertmh...@gmail.com wrote: ERROR:  constraints on permanent tables may reference only permanent tables HINT:  constraint %s Argh, hit send too soon. HINT: constraint %s references table %s nitpick That

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-06 Thread Robert Haas
On Fri, Apr 22, 2011 at 4:13 AM, Leonardo Francalanci m_li...@yahoo.it wrote: Maybe you should change  xl_act_commit to have a separate list of rels to drop the init fork for  (instead of mixing those with the list of files to drop as a  whole). I tried to follow your suggestion, thank you

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-05-06 Thread Robert Haas
On Fri, May 6, 2011 at 10:25 PM, Robert Haas robertmh...@gmail.com wrote: ERROR:  constraints on permanent tables may reference only permanent tables HINT:  constraint %s Argh, hit send too soon. HINT: constraint %s references table %s -- Robert Haas EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-22 Thread Leonardo Francalanci
Maybe you should change xl_act_commit to have a separate list of rels to drop the init fork for (instead of mixing those with the list of files to drop as a whole). I tried to follow your suggestion, thank you very much. Here's a first attempt at the patch. I tested it with: create

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-18 Thread Leonardo Francalanci
I think I coded a very basic version of the UNLOGGED to LOGGED patch (only wal_level=minimal case for the moment). To remove the INIT fork, I changed somehow PendingRelDelete to have a flag bool onlyInitFork so that the delete would remove only the INIT fork at commit. Everything works (note the

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-18 Thread Alvaro Herrera
Excerpts from Leonardo Francalanci's message of lun abr 18 09:36:13 -0300 2011: I think I coded a very basic version of the UNLOGGED to LOGGED patch (only wal_level=minimal case for the moment). To remove the INIT fork, I changed somehow PendingRelDelete to have a flag bool onlyInitFork so

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-16 Thread Leonardo Francalanci
If the master crashes while a transaction that used CREATE TABLE is unfinished, both the master and the standby will indefinitely retain identical, stray (not referenced by pg_class) files. The catalogs do reference the relfilenode of each unlogged relation; currently, that

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-11 Thread Leonardo Francalanci
But re-reading it, I don't understand: what's the difference in creating a new regular table and crashing before emitting the abort record, and converting an unlogged table to logged and crashing before emitting the abort record? How do the standby servers handle a CREATE TABLE

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-11 Thread Noah Misch
On Mon, Apr 11, 2011 at 11:41:18AM +0100, Leonardo Francalanci wrote: But re-reading it, I don't understand: what's the difference in creating a new regular table and crashing before emitting the abort record, and converting an unlogged table to logged and crashing before emitting

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-11 Thread Robert Haas
On Mon, Apr 11, 2011 at 10:29 AM, Noah Misch n...@leadboat.com wrote: On Mon, Apr 11, 2011 at 11:41:18AM +0100, Leonardo Francalanci wrote: But re-reading  it, I don't understand: what's the difference in creating a new  regular table and crashing before emitting the abort record, and  

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-10 Thread Robert Haas
On Sat, Apr 9, 2011 at 3:29 AM, Leonardo Francalanci m_li...@yahoo.it wrote: I'm pretty sure we wouldn't accept a patch for a  feature that would only work with wal_level=minimal, but it might be a useful  starting point for someone else to keep hacking on. I understand. Reading your post

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-09 Thread Leonardo Francalanci
I'm pretty sure we wouldn't accept a patch for a feature that would only work with wal_level=minimal, but it might be a useful starting point for someone else to keep hacking on. I understand. Reading your post at http://archives.postgresql.org/pgsql-hackers/2011-01/msg00315.php I thought

[HACKERS] switch UNLOGGED to LOGGED

2011-04-08 Thread Leonardo Francalanci
Hi, I read the discussion at http://archives.postgresql.org/pgsql-hackers/2011-01/msg00315.php From what I can understand, going from/to unlogged to/from logged in the wal_level == minimal case is not too complicated. Suppose I try to write a patch that allows ALTER TABLE tablename SET

Re: [HACKERS] switch UNLOGGED to LOGGED

2011-04-08 Thread Robert Haas
On Fri, Apr 8, 2011 at 6:01 AM, Leonardo Francalanci m_li...@yahoo.it wrote: I read the discussion at http://archives.postgresql.org/pgsql-hackers/2011-01/msg00315.php From what I can understand, going from/to unlogged to/from logged in the wal_level == minimal case is not too complicated.