Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-25 Thread Heikki Linnakangas
Simon Riggs wrote: On Sat, 2009-01-24 at 21:58 +0200, Heikki Linnakangas wrote: When replaying a DROP TABLE SPACE, you first try to remove the directory, and if that fails, you assume that it's because it's in use as a temp tablespace in a read-only transaction. That sounds like you think

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-25 Thread Grzegorz Jaskiewicz
On 2009-01-25, at 09:04, Simon Riggs wrote: On Sat, 2009-01-24 at 21:58 +0200, Heikki Linnakangas wrote: When replaying a DROP TABLE SPACE, you first try to remove the directory, and if that fails, you assume that it's because it's in use as a temp tablespace in a read-only transaction.

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-25 Thread Simon Riggs
On Sun, 2009-01-25 at 11:28 +0200, Heikki Linnakangas wrote: You then call ResolveRecoveryConflictWithVirtualXIDs to kill such transactions, and try removing the directory again. But ResolveRecoveryConflictWithVirtualXIDs doesn't wait for the target transaction to die anymore (or

Re: [HACKERS] Hot standby, conflict resolution

2009-01-25 Thread Simon Riggs
On Fri, 2009-01-23 at 21:30 +0200, Heikki Linnakangas wrote: Ok, then I think we have a little race condition. The startup process doesn't get any reply indicating that the target backend has processed the SIGINT and set the cached conflict LSN. The target backend might move ahead using

Re: [HACKERS] Hot standby, dropping a tablespace

2009-01-25 Thread Heikki Linnakangas
Simon Riggs wrote: 2. Kill all connections by given user. Hmm, not used for anything, actually. Should remove the roleId argument from GetConflictingVirtualXIDs. No, because we still need to add code to kill-connected-users if we drop role. Oh, I see, that's still a todo item. But we don't

Re: [HACKERS] Hot Standby (v9d)

2009-01-24 Thread Simon Riggs
On Sat, 2009-01-24 at 17:24 +1300, Mark Kirkwood wrote: version 9g - please use this for testing now I'm doing some test runs with this now. I notice an old flatfiles related bug has reappeared: I'm seeing an off-by-one error on xmax, in some cases. That then causes the flat file update

Re: [HACKERS] Hot Standby (v9d)

2009-01-24 Thread Simon Riggs
On Sat, 2009-01-24 at 11:20 +, Simon Riggs wrote: On Sat, 2009-01-24 at 17:24 +1300, Mark Kirkwood wrote: version 9g - please use this for testing now I'm doing some test runs with this now. I notice an old flatfiles related bug has reappeared: I'm seeing an off-by-one error

Re: [HACKERS] Hot Standby (v9d)

2009-01-24 Thread Mark Kirkwood
Simon Riggs wrote: On Sat, 2009-01-24 at 11:20 +, Simon Riggs wrote: On Sat, 2009-01-24 at 17:24 +1300, Mark Kirkwood wrote: version 9g - please use this for testing now I'm doing some test runs with this now. I notice an old flatfiles related bug has reappeared:

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Heikki Linnakangas
Simon Riggs wrote: * Put corrected version into GIT * Produce outstanding items as patch-on-patch via GIT I've applied the hot standby patch and recovery infra v9 patch to branches in my git repository, and pushed those to: git://git.postgresql.org/git/~hlinnaka/pgsql.git You can clone

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Simon Riggs
On Fri, 2009-01-23 at 16:14 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: * Put corrected version into GIT * Produce outstanding items as patch-on-patch via GIT I've applied the hot standby patch and recovery infra v9 patch to branches in my git repository, and pushed those to:

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-23 at 16:14 +0200, Heikki Linnakangas wrote: I made a couple of minor changes after importing your patches. I've applied them both to v9g, attached here. If you wouldn't mind redoing the initial step, I will promise not to do anything else to the code,

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Heikki Linnakangas
@@ -1601,6 +1602,24 @@ BufferProcessRecoveryConflictsIfAny(volatile BufferDesc *bufHdr) { XLogRecPtr bufLSN = BufferGetLSN(bufHdr); + /* +* If the buffer is recent we may need to cancel ourselves +* rather than risk

Re: [HACKERS] Hot standby, conflict resolution

2009-01-23 Thread Simon Riggs
On Fri, 2009-01-23 at 17:51 +0200, Heikki Linnakangas wrote: In ERROR mode, you don't really want to interrupt the target backend. In ReadBuffer, you're checking a global variable, BufferRecoveryConflictPending on each call, and if it's set, you check the buffer's LSN against the LSN of

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Simon Riggs
On Fri, 2009-01-23 at 18:22 +0200, Heikki Linnakangas wrote: @@ -1601,6 +1602,24 @@ BufferProcessRecoveryConflictsIfAny(volatile BufferDesc *bufHdr) { XLogRecPtr bufLSN = BufferGetLSN(bufHdr); + /* +* If the buffer is

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Heikki Linnakangas
Simon Riggs wrote: If you have a serializable transaction with subtransactions that suffers a serializability error it only cancels the current subtransaction. That means it's snapshot is still valid and can be used again. By analogy, as long as a transaction does not see any data that is

Re: [HACKERS] Hot standby, conflict resolution

2009-01-23 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-23 at 17:51 +0200, Heikki Linnakangas wrote: ISTM that if ReadBuffer read the value directly from the PGPROC entry, there would be no need for the signaling (in the ERROR mode). That is possible and I considered it. If we did it that way we would need to

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Alvaro Herrera
Heikki Linnakangas wrote: Simon Riggs wrote: If you have a serializable transaction with subtransactions that suffers a serializability error it only cancels the current subtransaction. That means it's snapshot is still valid and can be used again. By analogy, as long as a transaction does

Re: [HACKERS] Hot standby, conflict resolution

2009-01-23 Thread Simon Riggs
On Fri, 2009-01-23 at 21:30 +0200, Heikki Linnakangas wrote: Correct me if I'm wrong, but I thought the idea of this new conflict resolution was that the startup process doesn't need to wait for the target backend to die. Instead, the target backend knows to commit suicide if it

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Greg Stark
If you have a serializable transaction with subtransactions that suffers a serializability error it only cancels the current subtransaction. This is a complete tangent to your work, but I wonder if this is really right. I mean it's not as if we could have srrialized the transaction

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Jeff Davis
On Fri, 2009-01-23 at 20:13 +, Greg Stark wrote: If you have a serializable transaction with subtransactions that suffers a serializability error it only cancels the current subtransaction. This is a complete tangent to your work, but I wonder if this is really right. I mean it's

Re: [HACKERS] Hot Standby (v9d)

2009-01-23 Thread Mark Kirkwood
Simon Riggs wrote: On Thu, 2009-01-22 at 22:35 +, Simon Riggs wrote: * Bug fix v9 over next few days version 9g - please use this for testing now I'm doing some test runs with this now. I notice an old flatfiles related bug has reappeared: master: =# create database

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-20 Thread Heikki Linnakangas
Simon Riggs wrote: It would be useful to nibble away at the patch, committing all the necessary refactorings to make the patch work. That will reduce size of eventual commit. I committed this part now; one less thing to review. Please adjust the patch accordingly. -- Heikki Linnakangas

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-20 Thread Simon Riggs
On Tue, 2009-01-20 at 21:01 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: It would be useful to nibble away at the patch, committing all the necessary refactorings to make the patch work. That will reduce size of eventual commit. I committed this part now; one less thing to

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-20 Thread Heikki Linnakangas
Simon Riggs wrote: I did want you to commit those changes, but that was 8 days ago and much has changed since then. Now, I'm slightly worried that this may be a retrograde step. The last 3 days I've been refactoring the code to account for other requirements, as I have been discussing, and some

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 09:16 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2009-01-16 at 22:09 +0200, Heikki Linnakangas wrote: RecentGlobalXmin is just a hint, it lags behind the real oldest xmin that GetOldestXmin() would return. If another backend has a more recent

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 12:22 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: Well, steps 7 and 8 don't make sense. Your earlier comment was that it was possible for a WAL record to be written with a RecentGlobalXmin that was lower than other backends values. In step 9 the

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: One of us needs a coffee. Clearly, I put the kettle on... How does Transaction 4 have a RecentGlobalXmin of 2 in step (7), but at step (9) the value of RecentGlobalXmin has gone backwards? Looks like I mixed up the xids of the two transactions in steps 8 and 9. Let's

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 12:50 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: One of us needs a coffee. Clearly, I put the kettle on... I had one too, just in case. How does Transaction 4 have a RecentGlobalXmin of 2 in step (7), but at step (9) the value of RecentGlobalXmin has

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: Well, steps 7 and 8 don't make sense. Your earlier comment was that it was possible for a WAL record to be written with a RecentGlobalXmin that was lower than other backends values. In step 9 the RecentGlobalXmin is *not* lower than any other backend, it is the same. So if

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: GetSnapshotData() sets RecentGlobalXmin to the result of the snapshot's xmin. No. RecentGlobalXmin is set to the oldest *xmin* observed, across all running transactions. TransactionXmin is the xid of the oldest running transaction. IOW, RecentGlobalXmin is the xid of

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 14:00 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: GetSnapshotData() sets RecentGlobalXmin to the result of the snapshot's xmin. No. RecentGlobalXmin is set to the oldest *xmin* observed, across all running transactions. TransactionXmin is the xid of the

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: I suggest we handle this on the recovery side, not on the master, by deriving the xmin at the point the WAL record arrives. We would calculate it by looking at recovery procs only. That will likely give us a later value than we would get from the master, but that can't be

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 15:47 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: I suggest we handle this on the recovery side, not on the master, by deriving the xmin at the point the WAL record arrives. We would calculate it by looking at recovery procs only. That will likely give us a

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 12:54 +, Simon Riggs wrote: Some refactoring in this area is also required because we need to handle two other types of conflict to correctly support drop database and drop user, which is now underway. I've hung the drop database conflict code in dbase_redo(). For

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Heikki Linnakangas
Simon Riggs wrote: I prefer the last one, but if you think otherwise, please shout. We're now emitting WAL records for relcache invalidations, IIRC. I wonder if those are useful for this? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-19 Thread Simon Riggs
On Mon, 2009-01-19 at 16:50 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: I prefer the last one, but if you think otherwise, please shout. We're now emitting WAL records for relcache invalidations, IIRC. I wonder if those are useful for this? Tom already objected to putting strange

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-18 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-16 at 22:09 +0200, Heikki Linnakangas wrote: RecentGlobalXmin is just a hint, it lags behind the real oldest xmin that GetOldestXmin() would return. If another backend has a more recent RecentGlobalXmin value, and has killed more recent tuples on the page,

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-16 Thread Heikki Linnakangas
I don't think RecentGlobalXmin is good enough here: ! /* !* We would like to set an accurate latestRemovedXid, but there !* is no easy way of obtaining a useful value. So we use the !* probably far

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-16 Thread Simon Riggs
On Fri, 2009-01-16 at 16:33 +0200, Heikki Linnakangas wrote: I don't think RecentGlobalXmin is good enough here: ! /* !* We would like to set an accurate latestRemovedXid, but there !* is no easy way of obtaining a useful

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-16 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-16 at 16:33 +0200, Heikki Linnakangas wrote: I don't think RecentGlobalXmin is good enough here: ! /* !* We would like to set an accurate latestRemovedXid, but there !* is no easy way of

Re: [HACKERS] Hot Standby dev build (v8)

2009-01-16 Thread Simon Riggs
On Fri, 2009-01-16 at 22:09 +0200, Heikki Linnakangas wrote: RecentGlobalXmin is just a hint, it lags behind the real oldest xmin that GetOldestXmin() would return. If another backend has a more recent RecentGlobalXmin value, and has killed more recent tuples on the page, the

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Simon Riggs
On Fri, 2009-01-09 at 19:34 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: It would be useful to nibble away at the patch, committing all the necessary refactorings to make the patch work. That will reduce size of eventual commit. Agreed. This in particular is a change I feel

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Heikki Linnakangas
Simon Riggs wrote: The idea outlined before didn't deal with all call points for RecordIsCleanupRecord(), so doesn't actually work. Are we talking about the same thing? If we put the control of locking to the hands of the redo-function, I don't see why it couldn't use a lock of the right

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Heikki Linnakangas
Simon Riggs wrote: The idea outlined before didn't deal with all call points for RecordIsCleanupRecord(), so doesn't actually work. Hmm, grep finds two call points for that: ! case RECOVERY_TARGET_PAUSE_CLEANUP: ! /* !

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Simon Riggs
On Thu, 2009-01-15 at 18:05 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: The idea outlined before didn't deal with all call points for RecordIsCleanupRecord(), so doesn't actually work. Are we talking about the same thing? I guess not. Look at the other call points for that

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-15 Thread Simon Riggs
On Thu, 2009-01-15 at 18:16 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: Is the first really useful? I would understand advance until next cleanup record *that would block/kill queries*, but why would you want to advance until the next cleanup record? Minor difference there, but

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-15 Thread Simon Riggs
On Thu, 2009-01-08 at 22:31 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: If you want to do things a different way you need to say what you want to do and what effects those changes will have. I want to reduce the coupling between the primary and the master. The less they need to

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-15 Thread Simon Riggs
On Thu, 2009-01-15 at 22:10 +, Simon Riggs wrote: I notice that we are no longer able to record the databaseId for a connection, so query conflict resolution cannot be isolated to individual databases any longer. Wrong way round. Incoming WAL records from dbOid on them, so we can still

Re: [HACKERS] Hot standby and b-tree killed items

2009-01-13 Thread Simon Riggs
On Tue, 2008-12-30 at 18:31 +0200, Heikki Linnakangas wrote: You have to be careful to ignore the flags in read-only transactions that started in hot standby mode, even if recovery has since ended and we're in normal operation now. My initial implementation in v6 worked, but had a corner

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-13 Thread Simon Riggs
On Sat, 2009-01-10 at 09:40 +, Simon Riggs wrote: But looking at that, I think the 6a patch had a bug there: a subtransaction abort record would release locks for the whole top level xact. Fixed. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-12 Thread Simon Riggs
Heikki, can I get your feedback on this urgently please? I want to respond positively to your review comments and complete something you will find acceptable. But I need to know what that is, please. On Sun, 2009-01-11 at 11:55 +, Simon Riggs wrote: On Sun, 2009-01-11 at 10:41 +0200, Heikki

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-12 Thread Simon Riggs
On Mon, 2009-01-12 at 14:10 +0200, Heikki Linnakangas wrote: However, I think XactLockTableWait() doesn't need to know the parent either. (This feels more like wishful thinking, but here goes anyway). We release locks *after* TransactionIdAbortTree() has fully executed, so the test for

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-12 Thread Heikki Linnakangas
Simon Riggs wrote: Rather than store the parent xid itself we store the difference between the current xid and the parent xid. Typically this will be less than 65535; when it is not we set it to zero but issue an xid assignment xlog record. That sounds pretty hacky. However, I think

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-12 Thread Simon Riggs
Please commit soon On Fri, 2009-01-09 at 18:30 +0200, Heikki Linnakangas wrote: The hot standby patch has some hacks to decide which full-page-images can be restored holding an exclusive lock and which ones need a vacuum-strength lock. It's not very pretty as is, as mentioned in

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-11 Thread Heikki Linnakangas
Simon Riggs wrote: There's a confusion in the patch between top level xid and parent xid. The xl field is named parentxid but actually contains top level. That distinction is important because the code now uses the top level xid to locate the recovery proc, formerly the role of the slotid.

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-11 Thread Simon Riggs
On Sun, 2009-01-11 at 10:41 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: There's a confusion in the patch between top level xid and parent xid. The xl field is named parentxid but actually contains top level. That distinction is important because the code now uses the top level xid

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-10 Thread Simon Riggs
On Thu, 2009-01-08 at 20:30 +0200, Heikki Linnakangas wrote: Since I've been whining about that for some time, I figured I have to put my money where my mouth is, so here's a patch based on v6a that eliminates the concept of slotids, as well as the new xl_info2 field in XLogRecord. This

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-10 Thread Simon Riggs
On Sat, 2009-01-10 at 09:40 +, Simon Riggs wrote: This leads to an error when we SubTransSetParent(child_xid, top_xid); since this assumes that the top_xid is the parent, which it is not. Mostly you wouldn't notice unless you were looking up the subtrans status for an xid that had

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-10 Thread Simon Riggs
On Sat, 2009-01-10 at 11:45 +, Simon Riggs wrote: On Sat, 2009-01-10 at 09:40 +, Simon Riggs wrote: This leads to an error when we SubTransSetParent(child_xid, top_xid); since this assumes that the top_xid is the parent, which it is not. Mostly you wouldn't notice unless you were

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-09 Thread Simon Riggs
On Thu, 2009-01-08 at 15:50 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Thu, 2009-01-08 at 22:31 +0200, Heikki Linnakangas wrote: When a backend dies with FATAL, it writes an abort record before exiting. (I was under the impression it doesn't until few minutes

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-09 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-01-08 at 15:50 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Thu, 2009-01-08 at 22:31 +0200, Heikki Linnakangas wrote: When a backend dies with FATAL, it writes an abort record before exiting. (I was under the impression it doesn't until

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-09 Thread Simon Riggs
On Fri, 2009-01-09 at 12:33 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-01-08 at 15:50 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Thu, 2009-01-08 at 22:31 +0200, Heikki Linnakangas wrote: When a backend dies with FATAL, it writes an abort

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-09 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-09 at 12:33 +0200, Heikki Linnakangas wrote: A related issue is that currently the recovery PANICs if it runs out of recovery procs. I think that's not acceptable, regardless of whether we use slotids or some other method to avoid it in normal operation,

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-09 Thread Simon Riggs
On Fri, 2009-01-09 at 13:23 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2009-01-09 at 12:33 +0200, Heikki Linnakangas wrote: A related issue is that currently the recovery PANICs if it runs out of recovery procs. I think that's not acceptable, regardless of whether we

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-09 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-09 at 13:23 +0200, Heikki Linnakangas wrote: I mean the standby should stop trying to track the in progress transactions in recovery procs, and apply the WAL records like it does before the consistent state is reached. ... So, if we don't PANIC, how should

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-09 Thread Simon Riggs
On Fri, 2009-01-09 at 14:38 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2009-01-09 at 13:23 +0200, Heikki Linnakangas wrote: I mean the standby should stop trying to track the in progress transactions in recovery procs, and apply the WAL records like it does before the

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-09 Thread Simon Riggs
On Fri, 2009-01-09 at 18:30 +0200, Heikki Linnakangas wrote: The hot standby patch has some hacks to decide which full-page-images can be restored holding an exclusive lock and which ones need a vacuum-strength lock. It's not very pretty as is, as mentioned in comments too. Agreed! How

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-09 Thread Simon Riggs
On Fri, 2009-01-09 at 18:30 +0200, Heikki Linnakangas wrote: How about we refactor things? Was that a patch against HEAD or a patch on patch? It would be useful to nibble away at the patch, committing all the necessary refactorings to make the patch work. That will reduce size of eventual

Re: [HACKERS] Hot standby, RestoreBkpBlocks and cleanup locks

2009-01-09 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-01-09 at 18:30 +0200, Heikki Linnakangas wrote: How about we refactor things? Was that a patch against HEAD or a patch on patch? Against HEAD. It would be useful to nibble away at the patch, committing all the necessary refactorings to make the patch work.

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-09 Thread Simon Riggs
On Fri, 2009-01-09 at 11:20 +, Simon Riggs wrote: On Fri, 2009-01-09 at 12:33 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-01-08 at 15:50 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Thu, 2009-01-08 at 22:31 +0200, Heikki Linnakangas

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-08 Thread Simon Riggs
On Thu, 2009-01-08 at 20:30 +0200, Heikki Linnakangas wrote: I've mentioned before that I don't like the slotid stuff. From an architectural point of view, we should try to keep the extra communication between primary and standby to a minimum, for the sake of robustness. The primary

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-08 Thread Simon Riggs
On Thu, 2009-01-08 at 20:30 +0200, Heikki Linnakangas wrote: I've mentioned before that I don't like the slotid stuff. From an architectural point of view, we should try to keep the extra communication between primary and standby to a minimum, for the sake of robustness. The primary

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-08 Thread Heikki Linnakangas
Simon Riggs wrote: * if FATAL errors occur, yet we have long running transactions then we have no way of removing those entries from the recovery procs. Since we have a fixed pool of recovery transactions we won't have anywhere to store that data. Snapshot sizes are fixed maximum with

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-08 Thread Heikki Linnakangas
Simon Riggs wrote: If you want to do things a different way you need to say what you want to do and what effects those changes will have. I want to reduce the coupling between the primary and the master. The less they need to communicate, the better. I want to get rid of slotid, and as many

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-08 Thread Simon Riggs
On Thu, 2009-01-08 at 22:31 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: * if FATAL errors occur, yet we have long running transactions then we have no way of removing those entries from the recovery procs. Since we have a fixed pool of recovery transactions we won't have anywhere

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-08 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-01-08 at 22:31 +0200, Heikki Linnakangas wrote: When a backend dies with FATAL, it writes an abort record before exiting. (I was under the impression it doesn't until few minutes ago myself, when I actually read the shutdown code :-)) Not in all cases; keep

Re: [HACKERS] Hot standby, slot ids and stuff

2009-01-08 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Thu, 2009-01-08 at 22:31 +0200, Heikki Linnakangas wrote: When a backend dies with FATAL, it writes an abort record before exiting. (I was under the impression it doesn't until few minutes ago myself, when I actually read the shutdown code :-))

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-31 Thread Simon Riggs
On Tue, 2008-12-30 at 18:31 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: (a) always ignore LP_DEAD flags we see when reading index during recovery. This sounds simplest, and it's nice to not clear the flags for the benefit of transactions running after the recovery is done.

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-30 Thread Simon Riggs
On Fri, 2008-12-19 at 09:22 -0500, Greg Stark wrote: I'm confused shouldn't read-only transactions on the slave just be hacked to not set any hint bits including lp_delete? It seems there are multiple issues involved and I saw only the first of these initially. I want to explicitly separate

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-30 Thread Heikki Linnakangas
Simon Riggs wrote: Issues (2) and (3) would go away entirely if both standby and primary always had the same xmin value as a system-wide setting. i.e. the standby and primary are locked together at their xmins. Perhaps that was Heikki's intention in recent suggestions? No, I only suggested

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-30 Thread Heikki Linnakangas
Simon Riggs wrote: (a) always ignore LP_DEAD flags we see when reading index during recovery. This sounds simplest, and it's nice to not clear the flags for the benefit of transactions running after the recovery is done. You have to be careful to ignore the flags in read-only transactions

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-29 Thread Heikki Linnakangas
marcin mank wrote: Perhaps we should listen to the people that have said they don't want queries cancelled, even if the alternative is inconsistent answers. I don't like that much. PostgreSQL has traditionally avoided that very hard. It's hard to tell what kind of inconsistencies you'd get,

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-25 Thread marcin mank
Perhaps we should listen to the people that have said they don't want queries cancelled, even if the alternative is inconsistent answers. I think an alternative to that would be if the wal backlog is too big, let current queries finish and let incoming queries wait till the backlog gets

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Simon Riggs
On Tue, 2008-12-23 at 23:59 -0500, Robert Treat wrote: On Friday 19 December 2008 19:36:42 Simon Riggs wrote: Perhaps we should listen to the people that have said they don't want queries cancelled, even if the alternative is inconsistent answers. That is easily possible yet is not

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Pavan Deolasee
On Wed, Dec 24, 2008 at 4:41 PM, Simon Riggs si...@2ndquadrant.com wrote: Greg and Heikki have highlighted in this thread some aspects of btree garbage collection that will increase the chance of queries being cancelled in various circumstances Even HOT-prune may lead to frequent query

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 16:48 +0530, Pavan Deolasee wrote: On Wed, Dec 24, 2008 at 4:41 PM, Simon Riggs si...@2ndquadrant.com wrote: Greg and Heikki have highlighted in this thread some aspects of btree garbage collection that will increase the chance of queries being cancelled in

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Pavan Deolasee
On Wed, Dec 24, 2008 at 5:26 PM, Simon Riggs si...@2ndquadrant.com wrote: The patch does go to some trouble to handle that case, as I'm sure you've seen. Are you saying that part of the patch is ineffective and should be removed, or? Umm.. are you talking about the wait mechanism ? That's

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 17:56 +0530, Pavan Deolasee wrote: On Wed, Dec 24, 2008 at 5:26 PM, Simon Riggs si...@2ndquadrant.com wrote: The patch does go to some trouble to handle that case, as I'm sure you've seen. Are you saying that part of the patch is ineffective and should be

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Simon Riggs
On Wed, 2008-12-24 at 09:59 -0500, Robert Treat wrote: I think the uncertainty comes from peoples experience with typical replication use cases vs a lack of experience with this current implementation. Quite possibly. Publishing user feedback on this will be very important in making

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-24 Thread Pavan Deolasee
On Wed, Dec 24, 2008 at 7:18 PM, Simon Riggs si...@2ndquadrant.com wrote: With respect, I was hoping you might look in the patch and see if you agree with the way it is handled. No need to remember. The whole latestRemovedXid concept is designed to do help. Well, that's common for all

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-23 Thread Simon Riggs
On Sat, 2008-12-20 at 22:07 +0200, Heikki Linnakangas wrote: Gregory Stark wrote: A vacuum being replayed -- even in a different database -- could trigger the error. Or with the btree split issue, a data load -- again even in a different database -- would be quite likely cause your

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-23 Thread Simon Riggs
On Sat, 2008-12-20 at 20:09 -0300, Alvaro Herrera wrote: Heikki Linnakangas wrote: Gregory Stark wrote: A vacuum being replayed -- even in a different database -- could trigger the error. Or with the btree split issue, a data load -- again even in a different database -- would be

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-23 Thread Simon Riggs
On Fri, 2008-12-19 at 14:23 -0600, Kevin Grittner wrote: I guess making it that SQLSTATE would make it simpler to understand why the error occurs and also how to handle it (i.e. resubmit). Precisely. Just confirming I will implement the SQLSTATE as requested. I recognize my own and

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-23 Thread Robert Treat
On Saturday 20 December 2008 04:10:21 Heikki Linnakangas wrote: Simon Riggs wrote: On Sat, 2008-12-20 at 09:21 +0200, Heikki Linnakangas wrote: Gregory Stark wrote: Simon Riggs si...@2ndquadrant.com writes: Increasing the waiting time increases the failover time and thus decreases the

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-23 Thread Robert Treat
On Friday 19 December 2008 19:36:42 Simon Riggs wrote: Perhaps we should listen to the people that have said they don't want queries cancelled, even if the alternative is inconsistent answers. That is easily possible yet is not currently an option. Plus we have the option I referred to up

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-21 Thread Heikki Linnakangas
Alvaro Herrera wrote: Heikki Linnakangas wrote: Gregory Stark wrote: A vacuum being replayed -- even in a different database -- could trigger the error. Or with the btree split issue, a data load -- again even in a different database -- would be quite likely cause your SELECT to be killed.

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-21 Thread Alvaro Herrera
Heikki Linnakangas wrote: Alvaro Herrera wrote: Heikki Linnakangas wrote: Gregory Stark wrote: A vacuum being replayed -- even in a different database -- could trigger the error. Or with the btree split issue, a data load -- again even in a different database -- would be quite likely

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-20 Thread Simon Riggs
On Sat, 2008-12-20 at 09:21 +0200, Heikki Linnakangas wrote: Gregory Stark wrote: Simon Riggs si...@2ndquadrant.com writes: Increasing the waiting time increases the failover time and thus decreases the value of the standby as an HA system. Others value high availability higher than

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-20 Thread Heikki Linnakangas
Simon Riggs wrote: On Sat, 2008-12-20 at 09:21 +0200, Heikki Linnakangas wrote: Gregory Stark wrote: Simon Riggs si...@2ndquadrant.com writes: Increasing the waiting time increases the failover time and thus decreases the value of the standby as an HA system. Others value high availability

Re: [HACKERS] Hot standby and b-tree killed items

2008-12-20 Thread Heikki Linnakangas
Gregory Stark wrote: A vacuum being replayed -- even in a different database -- could trigger the error. Or with the btree split issue, a data load -- again even in a different database -- would be quite likely cause your SELECT to be killed. Hmm, I wonder if we should/could track the

<    4   5   6   7   8   9   10   >