Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Tom Lane wrote: There's no equivalent of XLogArchivingActive()? XLogArchivingMode() == false enables us to skip WAL-logging in operations like CLUSTER or COPY, which is a big optimization. I don't see anything like that in Hot

Re: [HACKERS] Hot standby and removing VACUUM FULL

2009-11-21 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Tom Lane wrote: There's no equivalent of XLogArchivingActive()? XLogArchivingMode() == false enables us to skip WAL-logging in operations like CLUSTER or COPY, which is a big optimization. I don't see anything

Re: [HACKERS] Hot standby, overflowed snapshots, testing

2009-11-15 Thread Simon Riggs
On Sat, 2009-11-14 at 08:43 -0800, Robert Hodges wrote: I can help set up automated basic tests for hot standby using 1+1 setups on Amazon. I¹m already working on tests for warm standby for our commercial Tungsten implementation and need to solve the problem of creating tests that adapt

Re: [HACKERS] Hot standby, race condition between recovery snapshot and commit

2009-11-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Sat, 2009-11-14 at 14:59 +0200, Heikki Linnakangas wrote: I can't see any obvious way around that. Huh? We're only doing this strict locking approach because you insisted that the looser approach was not acceptable. Take it easy, Simon. By obvious, I meant trivial or

Re: [HACKERS] Hot standby, race condition between recovery snapshot and commit

2009-11-15 Thread Heikki Linnakangas
Oh, forgot to mention another thing that I've been pondering: Currently, the running-xacts record is written to the WAL after the checkpoint record. There's a small chance that you get an xlog switch in between. If that happens, it might take a long time after the checkpoint record until the

Re: [HACKERS] Hot standby, race condition between recovery snapshot and commit

2009-11-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Sun, 2009-11-15 at 21:37 +0200, Heikki Linnakangas wrote: Am I missing anything? Will review. Thanks! Please use the head of git branch, I already found one major oversight in what I posted that's fixed there... I should go to bed already. -- Heikki Linnakangas

Re: [HACKERS] Hot standby, overflowed snapshots, testing

2009-11-15 Thread Robert Hodges
On 11/15/09 2:25 AM PST, Simon Riggs si...@2ndquadrant.com wrote: On Sat, 2009-11-14 at 08:43 -0800, Robert Hodges wrote: I can help set up automated basic tests for hot standby using 1+1 setups on Amazon. I¹m already working on tests for warm standby for our commercial Tungsten

Re: [HACKERS] Hot standby, overflowed snapshots, testing

2009-11-14 Thread Robert Hodges
Hi Simon and Heikki, I can help set up automated basic tests for hot standby using 1+1 setups on Amazon. I¹m already working on tests for warm standby for our commercial Tungsten implementation and need to solve the problem of creating tests that adapt flexibly across different replication

Re: [HACKERS] Hot standby, overflowed snapshots, testing

2009-11-13 Thread Simon Riggs
On Fri, 2009-11-13 at 22:19 +0200, Heikki Linnakangas wrote: I got the impression earlier that you had some test environment set up to test hot standby. Can you share any details of what test cases you've run? Fair question. The Sep 15 submission happened too quickly for us to mobilise

Re: [HACKERS] Hot Standby status

2009-11-11 Thread Heikki Linnakangas
Devrim GÜNDÜZ wrote: Is there a reason why recovery.conf.sample does not include (sample) entries for recovery_connections and max_standby_delay? No, they probably should be included. I'll add them, thanks. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via

Re: [HACKERS] Hot Standby status

2009-11-11 Thread Simon Riggs
On Tue, 2009-11-10 at 22:00 +0200, Devrim GÜNDÜZ wrote: On Tue, 2009-11-10 at 20:36 +0200, Heikki Linnakangas wrote: Attached is the latest and greatest patch against CVS head, taken from the hs-riggs branch in my git repository. Is there a reason why recovery.conf.sample does not include

Re: [HACKERS] Hot Standby status

2009-11-11 Thread Simon Riggs
On Tue, 2009-11-10 at 15:11 -0500, Michael Glaesemann wrote: I skimmed through the documentation to get a better handle on what this will mean. Thanks for this and any further corrections/additions. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Hot standby, freezing

2009-11-11 Thread Simon Riggs
On Wed, 2009-11-11 at 12:35 +0200, Heikki Linnakangas wrote: While reading through the patch for what must be the 100th time by now, :-) it occurred to me that this comment in heap_xlog_freeze: + /* +* Freezing tuples does not require conflict processing +*/ is

Re: [HACKERS] Hot Standby status

2009-11-10 Thread Devrim GÜNDÜZ
On Tue, 2009-11-10 at 20:36 +0200, Heikki Linnakangas wrote: Attached is the latest and greatest patch against CVS head, taken from the hs-riggs branch in my git repository. Is there a reason why recovery.conf.sample does not include (sample) entries for recovery_connections and

Re: [HACKERS] Hot Standby status

2009-11-10 Thread Michael Glaesemann
On Nov 10, 2009, at 13:36 , Heikki Linnakangas wrote: Attached is the latest and greatest patch against CVS head, taken from the hs-riggs branch in my git repository. Awesome. Thank you, Simon and Heikki! I skimmed through the documentation to get a better handle on what this will mean.

Re: [HACKERS] Hot Standby status

2009-11-10 Thread Heikki Linnakangas
Michael Glaesemann wrote: Other commands are in all-caps. Any reason INSERT, UPDATE, DELETE, TRUNCATE, CREATE, DROP, ALTER, and COMMENT are not? No. Thanks, I'll fix that. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Hot Standby and 64+ subxids (was COPY enhancements)

2009-11-09 Thread Andrew Dunstan
Simon Riggs wrote: On Thu, 2009-10-08 at 11:50 -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: Subcommitting every single row is going to be really painful, especially after Hot Standby goes in and we have to issue a WAL record after every 64 subtransactions (AIUI).

Re: [HACKERS] Hot Standby and 64+ subxids (was COPY enhancements)

2009-11-09 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Simon Riggs wrote: HS writes a WAL record for subtransactions at the point that the subxid cache overflows for any single transaction. Current cache size = 64. Top-level transaction then writes one additional WAL record every additional 64 subxids

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-22 Thread Simon Riggs
On Thu, 2009-10-22 at 07:55 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Wed, 2009-10-21 at 23:02 +0300, Heikki Linnakangas wrote: Hmm, dunno about that, but there is one problem with the grant to dummy proc, then release in startup process approach. What if there isn't enough

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-22 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-10-22 at 07:55 +0300, Heikki Linnakangas wrote: Making some effort to transfer locks instead of acquiring+releasing would eliminate the need for having extra lock space available when switching from hot standby mode to normal operation. This isn't very clear.

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-22 Thread Simon Riggs
On Thu, 2009-10-22 at 09:41 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-10-22 at 07:55 +0300, Heikki Linnakangas wrote: Making some effort to transfer locks instead of acquiring+releasing would eliminate the need for having extra lock space available when switching

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Simon Riggs
On Wed, 2009-10-21 at 19:37 +0300, Heikki Linnakangas wrote: So, I'm quite eager to just revert all those lock_twophase_recover() changes, and always rely on the grant lock to dummy proc, then release it in startup process method. If we don't want to rely on that, PostPrepare_Locks is an

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-10-21 at 19:37 +0300, Heikki Linnakangas wrote: So, I'm quite eager to just revert all those lock_twophase_recover() changes, and always rely on the grant lock to dummy proc, then release it in startup process method. If we don't want to rely on that,

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Simon Riggs
On Wed, 2009-10-21 at 23:02 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Wed, 2009-10-21 at 19:37 +0300, Heikki Linnakangas wrote: So, I'm quite eager to just revert all those lock_twophase_recover() changes, and always rely on the grant lock to dummy proc, then release it

Re: [HACKERS] Hot standby, prepared xacts, locks

2009-10-21 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-10-21 at 23:02 +0300, Heikki Linnakangas wrote: Hmm, dunno about that, but there is one problem with the grant to dummy proc, then release in startup process approach. What if there isn't enough shared memory available to re-acquire the lock for the dummy proc?

Re: [HACKERS] Hot standby, pausing recovery

2009-10-20 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-10-16 at 09:33 +0300, Heikki Linnakangas wrote: If you pause recovery, and then continue, we reset to target none mode, even if a stopping point was set previously. Yes, currently. Resetting the target mode will do what you want, rather than continue.

Re: [HACKERS] Hot standby, pausing recovery

2009-10-20 Thread Simon Riggs
On Tue, 2009-10-20 at 10:33 +0300, Heikki Linnakangas wrote: At this point, I'd like to cut out all those control functions to pause/stop at various points from the patch. OK -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Hot standby, pausing recovery

2009-10-20 Thread Simon Riggs
On Tue, 2009-10-20 at 09:43 +0100, Simon Riggs wrote: On Tue, 2009-10-20 at 10:33 +0300, Heikki Linnakangas wrote: At this point, I'd like to cut out all those control functions to pause/stop at various points from the patch. OK Maybe OK, I should say. Some parts are important for testing

Re: [HACKERS] Hot standby, pausing recovery

2009-10-16 Thread Simon Riggs
On Fri, 2009-10-16 at 09:33 +0300, Heikki Linnakangas wrote: In advance-mode, we will merrilly skip over a WAL record that's a recovery stop target. Is that a bug or a feature? Merrily?!? I saw it more as a sombre stepping motion. Advance currently means set target at next record and

Re: [HACKERS] Hot standby status

2009-10-15 Thread Simon Riggs
On Thu, 2009-10-15 at 10:33 +0300, Heikki Linnakangas wrote: There's been a lot of churn in hot standby since the beginning of the commitfest, so I thought it would be good to summarize where we are. Attached is the latest and greatest patch against CVS head, taken from the hs-riggs branch

Re: [HACKERS] Hot standby status

2009-10-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-10-15 at 10:33 +0300, Heikki Linnakangas wrote: Let me know if I'm missing something. And please feel free to help, by testing, by reviewing and commenting on the patch, or by addressing any of the above issues. I will continue working on this, but this is a big

Re: [HACKERS] Hot standby, recovery_starts_paused

2009-10-15 Thread Simon Riggs
On Fri, 2009-10-16 at 01:43 +0300, Heikki Linnakangas wrote: recovery_starts_paused is useless as it is. It pauses the recovery right after the first WAL record, all right, but before we see a running-xacts record, we won't let any backends in. And if you can't connect, you can't unpause, so

Re: [HACKERS] Hot standby, xlog_xact_assignment and unreported subxids.

2009-10-14 Thread Simon Riggs
On Wed, 2009-10-14 at 14:43 +0300, Heikki Linnakangas wrote: In the hot standby patch, we have this comment in procarray.c: It is * important that the XLOG_XACT_ASSIGNMENT record contain *all* subxids * not just those so far unreported because the sole purpose is to ensure * we can

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Bruce Momjian
Simon Riggs wrote: You have posted patches that I have said I don't agree with. My name is going to be on this when it goes in, so I don't think it makes any sense to force that commit to include changes I don't agree with. I cannot prevent you making changes afterwards, nor would I wish to.

Re: [HACKERS] Hot Standby on git

2009-10-09 Thread Heikki Linnakangas
While playing with conflict resolution, I bumped into this: postgres=# begin ISOLATION LEVEL SERIALIZABLE; BEGIN postgres=# SELECT * FROM foo; id | data +-- 12 | (1 row) postgres=# SELECT * FROM foo; id | data +-- 12 | (1 row) postgres=# SELECT * FROM foo; id | data

Re: [HACKERS] Hot Standby on git

2009-10-09 Thread Heikki Linnakangas
Simon Riggs wrote: Tom Lane wrote: [ scratches head ... ] Why is hot standby messing with this sort of thing at all? It sounds like a performance optimization that should be considered separately, and *later*. Yeah, I too considered just ripping it out. Simon is worried that locking all

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Robert Haas
On Oct 9, 2009, at 1:21 PM, Bruce Momjian br...@momjian.us wrote: Simon Riggs wrote: You have posted patches that I have said I don't agree with. My name is going to be on this when it goes in, so I don't think it makes any sense to force that commit to include changes I don't agree with.

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Andrew Dunstan
Robert Haas wrote: But at least for simple features I think that there would be a value in separating the patch author's work from the committer's adjustments. That is just going to make life harder for committers. There are plenty of things with my name on them that are not exactly

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Joshua D. Drake
On Fri, 2009-10-09 at 14:05 -0400, Andrew Dunstan wrote: Robert Haas wrote: But at least for simple features I think that there would be a value in separating the patch author's work from the committer's adjustments. That is just going to make life harder for committers. There

Re: [HACKERS] Hot Standby 0.2.1

2009-10-09 Thread Bruce Momjian
Andrew Dunstan wrote: Robert Haas wrote: But at least for simple features I think that there would be a value in separating the patch author's work from the committer's adjustments. That is just going to make life harder for committers. There are plenty of things with my name

Re: [HACKERS] Hot Standby 0.2.1

2009-10-08 Thread Simon Riggs
On Wed, 2009-10-07 at 23:19 -0400, Heikki Linnakangas wrote: Heikki Linnakangas wrote: I've made public the version I'm working on. That's the version I'm ultimately going to commit. It would be a lot more helpful if you provided these patches over that version. Otherwise I have to

Re: [HACKERS] Hot Standby 0.2.1

2009-10-08 Thread Heikki Linnakangas
Simon Riggs wrote: I would still like you to make a clear statement that the contents of that repository are BSD licenced open source contributions. Ok. All the content in the repository at git://git.postgresql.org/git/users/heikki/postgres.git is released under the same BSD license as

Re: [HACKERS] Hot Standby 0.2.1

2009-10-08 Thread Simon Riggs
On Thu, 2009-10-08 at 07:33 -0400, Heikki Linnakangas wrote: Simon Riggs wrote: I would still like you to make a clear statement that the contents of that repository are BSD licenced open source contributions. Ok. All the content in the repository at

Re: [HACKERS] Hot Standby 0.2.1

2009-10-08 Thread Simon Riggs
On Thu, 2009-10-08 at 12:49 +0100, Simon Riggs wrote: I'll get cracking on some changes. This will probably be next week now, just in case you're wondering when I'll start adding patches. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Hot Standby on git

2009-10-07 Thread Simon Riggs
On Fri, 2009-10-02 at 10:32 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I will docuemnt the recommendation to set max_standby_delay = 0 if performing an archive recovery (and explain why). Hmm, not sure if that's such a good piece of advice either. It will mean waiting for

Re: [HACKERS] Hot Standby 0.2.1

2009-10-07 Thread Simon Riggs
On Tue, 2009-09-22 at 12:53 +0300, Heikki Linnakangas wrote: It looks like the standby tries to remove XID 4323 from the known-assigned hash table, but it's not there because it was removed and set in pg_subtrans by an XLOG_XACT_ASSIGNMENT record earlier. I guess we should just not throw an

Re: [HACKERS] Hot Standby 0.2.1

2009-10-07 Thread Simon Riggs
On Wed, 2009-09-23 at 12:07 +0300, Heikki Linnakangas wrote: we need be careful to avoid putting any extra work into the normal recovery path. Otherwise bugs in hot standby related code can cause crash recovery to fail. Re-checked code and found a couple of additional places that needed tests

Re: [HACKERS] Hot Standby 0.2.1

2009-10-07 Thread Heikki Linnakangas
Simon Riggs wrote: On Tue, 2009-09-22 at 12:53 +0300, Heikki Linnakangas wrote: It looks like the standby tries to remove XID 4323 from the known-assigned hash table, but it's not there because it was removed and set in pg_subtrans by an XLOG_XACT_ASSIGNMENT record earlier. I guess we should

Re: [HACKERS] Hot Standby 0.2.1

2009-10-07 Thread Heikki Linnakangas
Heikki Linnakangas wrote: I've made public the version I'm working on. That's the version I'm ultimately going to commit. It would be a lot more helpful if you provided these patches over that version. Otherwise I have to refactor them over that codebase, possibly introducing new bugs.

Re: [HACKERS] Hot Standby on git

2009-10-06 Thread Simon Riggs
On Mon, 2009-10-05 at 18:30 -0400, Heikki Linnakangas wrote: Simon Riggs wrote: On Mon, 2009-09-28 at 11:25 +0300, Heikki Linnakangas wrote: Heikki Linnakangas wrote: Per Simon's request, for the benefit of the archive, here's all the changes I've done on the patch since he posted the

Re: [HACKERS] Hot Standby on git

2009-10-06 Thread Simon Riggs
On Tue, 2009-10-06 at 01:10 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: We discussed briefly your change 0011-Replace-per-proc-counters-of-loggable-locks-with-per.patch. I don't see how that helps at all. The objective of lock counters was to know if we can skip acquiring an

Re: [HACKERS] Hot Standby on git

2009-10-06 Thread Simon Riggs
On Sun, 2009-09-27 at 13:57 +0300, Heikki Linnakangas wrote: Per Simon's request, for the benefit of the archive, here's all the changes I've done on the patch since he posted the initial version for review for this commitfest as incremental patches. This is extracted from my git repository

Re: [HACKERS] Hot Standby on git

2009-10-06 Thread Simon Riggs
On Thu, 2009-10-01 at 18:47 +0300, Heikki Linnakangas wrote: And if you could please review the changes I've been doing, just to make sure I haven't inadvertently introduced new bugs. That has happened before, as you've rightfully reminded me :-). You posted 17 patches here. I've

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Simon Riggs
On Mon, 2009-09-28 at 11:25 +0300, Heikki Linnakangas wrote: Heikki Linnakangas wrote: Per Simon's request, for the benefit of the archive, here's all the changes I've done on the patch since he posted the initial version for review for this commitfest as incremental patches. This is

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Simon Riggs
On Sun, 2009-09-27 at 13:57 +0300, Heikki Linnakangas wrote: Per Simon's request, for the benefit of the archive, here's all the changes I've done on the patch since he posted the initial version for review for this commitfest as incremental patches. This is extracted from my git repository

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: I don't see how that helps at all. The objective of lock counters was to know if we can skip acquiring an LWlock on all lock partitions. This change keeps the lock counters yet acquires the locks we were trying to avoid. This change needs some

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Simon Riggs
On Mon, 2009-10-05 at 10:19 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: I don't see how that helps at all. The objective of lock counters was to know if we can skip acquiring an LWlock on all lock partitions. This change keeps the lock counters yet acquires the locks

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Heikki Linnakangas
Simon Riggs wrote: We discussed briefly your change 0011-Replace-per-proc-counters-of-loggable-locks-with-per.patch. I don't see how that helps at all. The objective of lock counters was to know if we can skip acquiring an LWlock on all lock partitions. This change keeps the lock counters

Re: [HACKERS] Hot Standby on git

2009-10-05 Thread Heikki Linnakangas
Simon Riggs wrote: On Mon, 2009-09-28 at 11:25 +0300, Heikki Linnakangas wrote: Heikki Linnakangas wrote: Per Simon's request, for the benefit of the archive, here's all the changes I've done on the patch since he posted the initial version for review for this commitfest as incremental

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: @@ -7061,6 +7061,15 @@ ShutdownXLOG(int code, Datum arg) else { /* +* Take a snapshot of running transactions and write this to WAL. +* This allows us to reconstruct the state of running transactions +* during archive recovery, if

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: I will docuemnt the recommendation to set max_standby_delay = 0 if performing an archive recovery (and explain why). Hmm, not sure if that's such a good piece of advice either. It will mean waiting for queries forever, which probably isn't what you want if you're performing

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 10:04 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: @@ -7061,6 +7061,15 @@ ShutdownXLOG(int code, Datum arg) else { /* +* Take a snapshot of running transactions and write this to WAL. +* This allows us to reconstruct the

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-10-02 at 10:04 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: @@ -7061,6 +7061,15 @@ ShutdownXLOG(int code, Datum arg) else { /* +* Take a snapshot of running transactions and write this to WAL. +* This allows us to

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 10:32 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I will docuemnt the recommendation to set max_standby_delay = 0 if performing an archive recovery (and explain why). Hmm, not sure if that's such a good piece of advice either. It will mean waiting for

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 10:43 +0300, Heikki Linnakangas wrote: It seems dangerous to write a WAL record after the shutdown checkpoint. It will be overwritten by subsequent startup, which is a recipe for trouble. I've said its a corner case and not worth spending time on. I'm putting it in at

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-10-02 at 10:43 +0300, Heikki Linnakangas wrote: It seems dangerous to write a WAL record after the shutdown checkpoint. It will be overwritten by subsequent startup, which is a recipe for trouble. I've said its a corner case and not worth spending time on.

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 11:26 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2009-10-02 at 10:43 +0300, Heikki Linnakangas wrote: It seems dangerous to write a WAL record after the shutdown checkpoint. It will be overwritten by subsequent startup, which is a recipe for

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 11:26 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2009-10-02 at 10:43 +0300, Heikki Linnakangas wrote: It seems dangerous to write a WAL record after the shutdown checkpoint. It will be overwritten by subsequent startup, which is a recipe for

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: I'd rather just skip this for now. It's a minor case anyway and there's nothing stopping writing their own RunningXactData records with a function, if it is needed. I can add a function for that. That won't help. There's no way to have it in a right place wrt. the shutdown

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 13:52 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I'd rather just skip this for now. It's a minor case anyway and there's nothing stopping writing their own RunningXactData records with a function, if it is needed. I can add a function for that. That won't

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Heikki Linnakangas
Simon Riggs wrote: I will add code to make a shutdown checkpoint be a valid starting place for Hot Standby, as long as there are no in-doubt prepared transactions. That way we know there are no xids still running and no locks, without needing to write a record to say so. Ok, I can live with

Re: [HACKERS] Hot Standby 0.2.1

2009-10-02 Thread Simon Riggs
On Sun, 2009-09-27 at 14:59 +0300, Heikki Linnakangas wrote: The locking in smgr_redo_commit and smgr_redo_abort doesn't look right. First of all, smgr_redo_abort is not holding XidGenLock and ProcArrayLock while modifying ShmemVariableCache-nextXid and ShmemVariableCache-latestCompletedXid,

Re: [HACKERS] Hot Standby on git

2009-10-02 Thread Simon Riggs
On Fri, 2009-10-02 at 18:04 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I will add code to make a shutdown checkpoint be a valid starting place for Hot Standby, as long as there are no in-doubt prepared transactions. That way we know there are no xids still running and no locks,

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Heikki Linnakangas
+ /* +* If our initial RunningXactData had an overflowed snapshot then we +* knew we were missing some subxids from our snapshot. We can use +* this data as an initial snapshot, but we cannot yet mark it valid. +* We know that the missing subxids are equal to or earlier than +

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Wed, 2009-09-30 at 18:45 +0300, Heikki Linnakangas wrote: Regarding this in InitStandbyDelayTimers: + /* +* If replication delay is enormously huge, just treat that as +* zero and work up from there. This prevents us from acting +* foolishly when replaying old log files. +

Re: [HACKERS] Hot Standby 0.2.1

2009-10-01 Thread Simon Riggs
On Sun, 2009-09-27 at 15:35 +0300, Heikki Linnakangas wrote: TransactionIdIsInProgress() doesn't consult the known-assigned-xids structure. That's a problem: in the standby, TransactionIdIsInProgress() can return false for a transaction that is still running in the master. HeapTupleSatisfies*

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Thu, 2009-10-01 at 14:29 +0300, Heikki Linnakangas wrote: + /* +* If our initial RunningXactData had an overflowed snapshot then we +* knew we were missing some subxids from our snapshot. We can use +* this data as an initial snapshot, but we cannot yet mark it valid. +*

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Wed, 2009-09-30 at 09:33 +0300, Heikki Linnakangas wrote: Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquire first before changing anything. Why? Looking at the other functions in that

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-09-30 at 09:33 +0300, Heikki Linnakangas wrote: Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquire first before changing anything. Why? Looking at the other

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Heikki Linnakangas
Simon Riggs wrote: Hmm, yes. ISTM that I'm still calculating latestRunningXid the old way while assuming it is calculated the new way. The new way is just to grab nextXid since we have XidGenLock and do TransactionIdRetreat() on it. Ok, good, that's what I thought too. I'll fix that. --

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Thu, 2009-10-01 at 18:47 +0300, Heikki Linnakangas wrote: Could you look into these two TODO items you listed on the wiki page: Unless we agree otherwise, if its listed on the Wiki page then I will work on it. Maybe not as when you might like it, but I am working through the list. 5 new

Re: [HACKERS] Hot Standby on git

2009-10-01 Thread Simon Riggs
On Thu, 2009-10-01 at 18:48 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: Hmm, yes. ISTM that I'm still calculating latestRunningXid the old way while assuming it is calculated the new way. The new way is just to grab nextXid since we have XidGenLock and do TransactionIdRetreat() on

Re: [HACKERS] Hot Standby on git

2009-09-30 Thread Heikki Linnakangas
Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquire first before changing anything. Why? Looking at the other functions in that file, all others that access both files are happy to acquire one

Re: [HACKERS] Hot Standby on git

2009-09-30 Thread Heikki Linnakangas
Looking at the changes to StartupMultiXact, you're changing the locking so that both MultiXactOffsetControlLock and MultiXactMemberControlLock are acquired first before changing anything. Why? Looking at the other functions in that file, all others that access both files are happy to acquire one

Re: [HACKERS] Hot Standby on git

2009-09-30 Thread Heikki Linnakangas
Regarding this in InitStandbyDelayTimers: + /* +* If replication delay is enormously huge, just treat that as +* zero and work up from there. This prevents us from acting +* foolishly when replaying old log files. +*/ + if (*currentDelay_ms 0) + *currentDelay_ms = 0; +

Re: [HACKERS] Hot Standby on git

2009-09-28 Thread Heikki Linnakangas
Heikki Linnakangas wrote: Per Simon's request, for the benefit of the archive, here's all the changes I've done on the patch since he posted the initial version for review for this commitfest as incremental patches. This is extracted from my git repository at

Re: [HACKERS] Hot Standby on git

2009-09-27 Thread Heikki Linnakangas
Per Simon's request, for the benefit of the archive, here's all the changes I've done on the patch since he posted the initial version for review for this commitfest as incremental patches. This is extracted from my git repository at git://git.postgresql.org/git/users/heikki/postgres.git. --

Re: [HACKERS] Hot Standby 0.2.1

2009-09-27 Thread Heikki Linnakangas
The locking in smgr_redo_commit and smgr_redo_abort doesn't look right. First of all, smgr_redo_abort is not holding XidGenLock and ProcArrayLock while modifying ShmemVariableCache-nextXid and ShmemVariableCache-latestCompletedXid, respectively, like smgr_redo_commit is. Attached patch fixes that.

Re: [HACKERS] Hot Standby 0.2.1

2009-09-27 Thread Heikki Linnakangas
TransactionIdIsInProgress() doesn't consult the known-assigned-xids structure. That's a problem: in the standby, TransactionIdIsInProgress() can return false for a transaction that is still running in the master. HeapTupleSatisfies* functions can incorrectly set HEAP_XMIN/XMAX_INVALID hint bits

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Robert Haas
On Sat, Sep 26, 2009 at 5:49 AM, Simon Riggs si...@2ndquadrant.com wrote: Just a note to say that Hot Standby patch is now on git repository  git://git.postgresql.org/git/users/simon/postgres Branch name: hot_standby Awesome! Thanks for taking the time to get this set up. The complete

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Simon Riggs
On Sat, 2009-09-26 at 09:29 -0400, Robert Haas wrote: I estimate that making the remaining changes noted on the Wiki and fully testing them will take at least 2 weeks. Gabriele Bartolini is assisting in this area, though neither of us are able to work full time on this. We still have

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Mark Mielke
On 09/26/2009 10:04 AM, Simon Riggs wrote: If you think there's something useful I could do, let me know and I'll take a look. I feel like I need a better way of unit testing new code. Some of the code in the patch is to handle corner cases, so recreating them is fairly hard. It is a

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Dan Colish
On Sat, Sep 26, 2009 at 10:45:17AM -0400, Mark Mielke wrote: On 09/26/2009 10:04 AM, Simon Riggs wrote: If you think there's something useful I could do, let me know and I'll take a look. I feel like I need a better way of unit testing new code. Some of the code in the patch is to handle

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Josh Berkus
I feel like I need a better way of unit testing new code. Some of the code in the patch is to handle corner cases, so recreating them is fairly hard. It is a nagging feeling that I am missing some knowledge here and would welcome some insight, or research, into better ways of doing general

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Mark Mielke
On 09/26/2009 02:28 PM, Dan Colish wrote: There are a variety of projects dedicated to creating C unit test frameworks. I don't have a lot of experience with them, but I have heard good things about check and cunit. Here's a link I found with a longer list of frameworks.

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread David E. Wheeler
On Sep 26, 2009, at 12:33 PM, Josh Berkus wrote: There's always pgtap. Whenever we find a new corner case, we add it to the development test suite. Also, for C TAP, there's [libtap](http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap ). You can then use `prove` which you likely already

Re: [HACKERS] Hot Standby on git

2009-09-26 Thread Alvaro Herrera
Mark Mielke escribió: Most real life code gets a little more complicated. For example, what if we want to simulate a network failure or out of disk space condition? What if we want to test out what happens when the Y2038 date is reached? This requires either complex test case setup that is

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Heikki Linnakangas
Looking at the startup sequence now. I see that you modified ExtendSUBTRANS so that it doesn't wipe out previously set values if it's called with out-of-order xids. I guess that works, although I think it can leave pages unzeroed if it's called with a large enough gap between xids, so that the

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Thu, 2009-09-24 at 13:33 +0300, Heikki Linnakangas wrote: Heikki Linnakangas wrote: The problem becomes a lot easier if we accept that it's OK to have a lock included in the running-xacts snapshot and also appear in a XLOG_RELATION_LOCK record later. The standby should handle that

Re: [HACKERS] Hot Standby 0.2.1

2009-09-25 Thread Simon Riggs
On Wed, 2009-09-23 at 19:07 +0300, Heikki Linnakangas wrote: Rather than keep the numHeldLocks counters per-proc in proc array, I think it would be simpler to have a single (or one per lock partition) counter in shared memory in lock.c. It's just an optimization to make it faster to find out

<    1   2   3   4   5   6   7   8   9   10   >