Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Florian G. Pflug
Tom Lane wrote: Florian G. Pflug [EMAIL PROTECTED] writes: And I'm quite tempted to not flush the XLOG at all during ABORT, and to only force synchronous commits if one of the to-be-deleted files is non-temporary. +1 on the first, but -1 on the second, because we'd have to track whether

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Gregory Stark
Florian G. Pflug [EMAIL PROTECTED] writes: It seems doable, but it's not pretty. One possible scheme would be to emit a record *after* chosing a name but *before* creating the file, No, because the way you know the name is good is a successful open(O_CREAT). The idea was to log *twice*.

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: Tom Lane wrote: +1 on the first, but -1 on the second, because we'd have to track whether deleted files are temp or not ... it's very unclear that it'd be worth the trouble. I'm confused. smgrcreate already tracks if it created a file for a temp

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Florian G. Pflug
Tom Lane wrote: Florian G. Pflug [EMAIL PROTECTED] writes: Tom Lane wrote: +1 on the first, but -1 on the second, because we'd have to track whether deleted files are temp or not ... it's very unclear that it'd be worth the trouble. I'm confused. smgrcreate already tracks if it created a

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: So should we not force synchronous commit if all to-be-deleted files are temporary? Or is that pushing our luck too much, because it opens the window of possible file leakage considerably wider? I think this area is something to be left for later.

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Simon Riggs
On Thu, 2007-08-30 at 00:33 -0400, Tom Lane wrote: Florian G. Pflug [EMAIL PROTECTED] writes: And I'm quite tempted to not flush the XLOG at all during ABORT, and to only force synchronous commits if one of the to-be-deleted files is non-temporary. +1 on the first, but -1 on the second,

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Simon Riggs
On Thu, 2007-08-30 at 14:14 -0400, Tom Lane wrote: Florian G. Pflug [EMAIL PROTECTED] writes: So should we not force synchronous commit if all to-be-deleted files are temporary? Or is that pushing our luck too much, because it opens the window of possible file leakage considerably wider?

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: -1 to second: If there are any temporary deleted files then it is because we've done a large sort, hash join etc, so avoiding the sync makes no difference to the overall response time. I think you're confused, actually: this is not about temporary sort

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Alvaro Herrera
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: -1 to second: If there are any temporary deleted files then it is because we've done a large sort, hash join etc, so avoiding the sync makes no difference to the overall response time. I think you're confused, actually: this is not

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: Why don't we create temp tables in a separate directory, anyway? That would make it trivially easy to get rid of them during recovery: just zap all the files in there. Hmm ... doesn't do anything to fix the problem of file leakage for crashes when

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-30 Thread Florian G. Pflug
Tom Lane wrote: Whether that's common enough to be worth special optimization, I'm not sure. Your point that the transaction may have been heavyweight enough that it'd hardly matter is still on-target, you just got there by the wrong reasoning ;-) My point is that there is hardly any effort

[HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-29 Thread Florian G. Pflug
Hi When we lazily assign XIDs, we gain another flag beside the existing MyXactMadeTempRelUpdates, MyXactMadeXLogEntry, MyLastRecPtr and smgr's pendingDeletes to tell what kind of actions a transaction performed. Adding TransactionIsIsValid(GetCurrentTransactionIdIfAny()) on top of that makes

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-29 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: I propose to do the following in my lazy XID assignment patch - can anyone see a hole in that? Cleaning up this area seems like a good idea. Just FYI, one reason why there are so many LastRec pointer variables is that the WAL record format used to

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-29 Thread Florian G. Pflug
Tom Lane wrote: Florian G. Pflug [EMAIL PROTECTED] writes: I propose to do the following in my lazy XID assignment patch - can anyone see a hole in that? One comment is that at the time we make an entry into smgr's pending-deletes list, I think we might not have acquired an XID yet --- if I

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-29 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: Tom Lane wrote: One thought here is that it's not clear that we really need a concept of transaction-controlled vs not-transaction-controlled xlog records anymore. I've thinking about keeping XLOG_NO_TRAN, and doing if (!no_tran)

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-29 Thread Florian G. Pflug
Tom Lane wrote: Florian G. Pflug [EMAIL PROTECTED] writes: Tom Lane wrote: One thought here is that it's not clear that we really need a concept of transaction-controlled vs not-transaction-controlled xlog records anymore. I've thinking about keeping XLOG_NO_TRAN, and doing if (!no_tran)

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-29 Thread Florian G. Pflug
Tom Lane wrote: One comment is that at the time we make an entry into smgr's pending-deletes list, I think we might not have acquired an XID yet --- if I understand your patch correctly, a CREATE TABLE would acquire an XID when it makes its first catalog insertion, and that happens after

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-29 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: One comment is that at the time we make an entry into smgr's pending-deletes list, I think we might not have acquired an XID yet Hm.. I was just going to implement this, but I'm now wondering if thats really worth it. Basically what you'd give up is

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-29 Thread Florian G. Pflug
Tom Lane wrote: Florian G. Pflug [EMAIL PROTECTED] writes: One comment is that at the time we make an entry into smgr's pending-deletes list, I think we might not have acquired an XID yet Hm.. I was just going to implement this, but I'm now wondering if thats really worth it. Basically

Re: [HACKERS] Advice on MyXactMade* flags, MyLastRecPtr, pendingDeletes and lazy XID assignment

2007-08-29 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: And I'm quite tempted to not flush the XLOG at all during ABORT, and to only force synchronous commits if one of the to-be-deleted files is non-temporary. +1 on the first, but -1 on the second, because we'd have to track whether deleted files are temp