Re: [HACKERS] Synchronous Log Shipping Replication

2008-09-19 Thread Simon Riggs
On Tue, 2008-09-09 at 09:11 +0100, Simon Riggs wrote: This gives us the Group Commit feature also, even if we are not using replication. So we can drop the commit_delay stuff. XLogBackgroundFlush() processes data page at a time if it can. That may not be the correct batch size

Re: [HACKERS] Assert Levels

2008-09-20 Thread Simon Riggs
that as an argument against doing something to enable the lighter checks in certain circumstances in the future. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Assert Levels

2008-09-20 Thread Simon Riggs
is invisible. If it is we throw an ERROR. But we already did that earlier. Now I've never seen that ERROR reported anywhere, so I'm thinking that I'd like to downgrade that somehow, yet still retain the ability to check it when things go strange. There are a few other examples. -- Simon Riggs www

Re: [HACKERS] Assert Levels

2008-09-20 Thread Simon Riggs
On Sat, 2008-09-20 at 11:28 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Fri, 2008-09-19 at 17:47 -0400, Tom Lane wrote: Well, there are certain things that --enable-cassert turns on that are outrageously expensive; notably CLOBBER_FREED_MEMORY

Re: [HACKERS] Foreign key constraint for array-field?

2008-09-21 Thread Simon Riggs
. Implementation would be fairly straightforward. ri_triggers currently assumes a non-array value is being checked, but that could be changed to IN(array). Multi-column keys with arrays sound confusing though. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent

Re: [HACKERS] Foreign key constraint for array-field?

2008-09-21 Thread Simon Riggs
On Sun, 2008-09-21 at 15:07 -0400, Andrew Dunstan wrote: Simon Riggs wrote: No, its not possible. Need a trigger. I think we should support it though. If we extend the relational model with arrays then it would be sensible if we support this aspect as well. Implementation would

Re: [HACKERS] Toasted table not deleted when no out of line columns left

2008-09-22 Thread Simon Riggs
this anyway for his customer, so it seems worth defining how it should look so we can accept it into core. VACUUM TOAST perhaps? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Toasted table not deleted when no out of line columns left

2008-09-22 Thread Simon Riggs
On Mon, 2008-09-22 at 10:59 +0200, Hans-Jürgen Schönig wrote: On Sep 22, 2008, at 9:46 AM, Simon Riggs wrote: I thought Hans meant cleanup, not drop? we definitely have to do something about this problem. I think the issue is identifying the problem. Reading the title of the post, I think

Re: [HACKERS] Initial prefetch performance testing

2008-09-22 Thread Simon Riggs
in the future. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] parallel pg_restore

2008-09-22 Thread Simon Riggs
the number of jobs that can run simultaneously. +1 -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] parallel pg_restore

2008-09-22 Thread Simon Riggs
memory and force additional I/O as a result. We might need a setting for total memory available, so pg_restore can try not to run tasks that will exceed that across settings. Preferably this wouldn't be just a pg_restore setting. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training

Re: [HACKERS] Toasted table not deleted when no out of line columns left

2008-09-22 Thread Simon Riggs
On Mon, 2008-09-22 at 07:53 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: I think the issue is identifying the problem. Reading the title of the post, I think Tom says no to *deleting* the toast table. He also says no to cleaning the table as part of DROP COLUMN. That still

Re: [HACKERS] parallel pg_restore

2008-09-22 Thread Simon Riggs
On Mon, 2008-09-22 at 11:38 -0400, Andrew Dunstan wrote: Simon Riggs wrote: On Mon, 2008-09-22 at 09:53 +0200, Dimitri Fontaine wrote: My intention is to have single-thread restore remain the default, at least for this go round, and have the user be able to choose --multi-thread

Re: [HACKERS] Initial prefetch performance testing

2008-09-22 Thread Simon Riggs
On Mon, 2008-09-22 at 16:46 +0100, Gregory Stark wrote: Simon Riggs [EMAIL PROTECTED] writes: On Mon, 2008-09-22 at 04:57 -0400, Greg Smith wrote: -As Greg Stark suggested, the larger the spindle count the larger the speedup, and the larger the prefetch size that might make sense

Re: [HACKERS] parallel pg_restore

2008-09-22 Thread Simon Riggs
On Mon, 2008-09-22 at 09:30 -0700, Joshua Drake wrote: On Mon, 22 Sep 2008 17:24:28 +0100 Simon Riggs [EMAIL PROTECTED] wrote: More importantly, I'm not convinced it's a good idea. It seems more like a footgun that will potentially try to launch thousands of simultaneous restore

Re: [HACKERS] Initial prefetch performance testing

2008-09-22 Thread Simon Riggs
would guess it won't be and you're left with a name more misleading than useful. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] get_relation_stats_hook()

2008-09-22 Thread Simon Riggs
your raise it. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] get_relation_stats_hook()

2008-09-22 Thread Simon Riggs
On Mon, 2008-09-22 at 18:41 +0100, Gregory Stark wrote: The easiest way Did you have further review comments? If so, I'll wait for those before making further mods. Thanks for ones so far. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-22 Thread Simon Riggs
On Thu, 2008-09-18 at 10:09 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: Do we really need a checkpoint there at all? Timelines only change at shutdown checkpoints. Hmm. I *think* that that is just a debugging

Re: [HACKERS] FSM, now without WAL-logging

2008-09-22 Thread Simon Riggs
that this might not be enough... I hadn't realised you would remove it completely. Did you identify WAL as the bottleneck? Is there some mid-way point between every time and almost never (VACUUM!)? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support

Re: [HACKERS] parallel pg_restore

2008-09-23 Thread Simon Riggs
set memory available across all threads with a single total value. I can live with jobs or multi-threads also, whichever we decide. Neither one is confusing to explain. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-23 Thread Simon Riggs
On Mon, 2008-09-22 at 23:06 +0100, Simon Riggs wrote: On Thu, 2008-09-18 at 10:09 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: Do we really need a checkpoint there at all? Timelines only change at shutdown

[HACKERS] Hot Standby Design

2008-09-23 Thread Simon Riggs
related to them. Comments or questions welcome here, or I will discuss specific areas in more detail as I tackle those topics. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] parallel pg_restore

2008-09-23 Thread Simon Riggs
On Tue, 2008-09-23 at 12:43 -0700, Joshua Drake wrote: On Tue, 23 Sep 2008 08:44:19 +0100 Simon Riggs [EMAIL PROTECTED] wrote: On Mon, 2008-09-22 at 15:05 -0400, Andrew Dunstan wrote: j and m happen to be two of those that are available. I honestly don't have a terribly

Re: [HACKERS] EXEC_BACKEND

2008-09-23 Thread Simon Riggs
On Tue, 2008-09-23 at 16:35 -0400, Bruce Momjian wrote: Simon Riggs wrote: I can't find anything coherent in docs/readme/comments to explain why it exists and what its implications are. It exists because Windows doesn't have fork(), only the equivalent of fork-and-exec

Re: [HACKERS] parallel pg_restore

2008-09-23 Thread Simon Riggs
On Tue, 2008-09-23 at 16:50 -0400, Andrew Dunstan wrote: If we get all that done by November we'll have done well. And we know that in some cases just this much can lead to reductions in restore time of the order of 80%. Agreed. Go for it. -- Simon Riggs www.2ndQuadrant.com

Re: [HACKERS] Subtransaction commits and Hot Standby

2008-09-23 Thread Simon Riggs
On Thu, 2008-09-18 at 15:59 +0100, Simon Riggs wrote: On Tue, 2008-09-16 at 10:11 -0400, Alvaro Herrera wrote: I wonder if the improved clog API required to mark multiple transactions as committed at once would be also useful to TransactionIdCommitTree which is used in regular

Re: [HACKERS] parallel pg_restore

2008-09-24 Thread Simon Riggs
On Tue, 2008-09-23 at 22:17 -0700, Joshua D. Drake wrote: Simon Riggs wrote: On Tue, 2008-09-23 at 16:50 -0400, Andrew Dunstan wrote: If we get all that done by November we'll have done well. And we know that in some cases just this much can lead to reductions in restore time

Re: [HACKERS] Hot Standby Design

2008-09-24 Thread Simon Riggs
is a Clone and not a Slave. A Slave is a separate database that is forced to duplicate the actions of the Master. The Standby is an exact copy, in every detail that matters. I can see it might be desirable to have it work that way, but that's not going to happen in the first release. -- Simon

Re: [HACKERS] Hot Standby Design

2008-09-24 Thread Simon Riggs
on a Standby node, but I think if I say there aren't any needed at all that would likely be wrong. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [PATCHES] [HACKERS] Subtransaction commits and Hot Standby

2008-09-24 Thread Simon Riggs
On Tue, 2008-09-23 at 22:47 +0100, Simon Riggs wrote: I've tested this some more and am much happier with it now. The concept is fine, but I've found a coding bug in further testing. Please wait now for new version before review. -- Simon Riggs www.2ndQuadrant.com PostgreSQL

Re: [HACKERS] Initial prefetch performance testing

2008-09-24 Thread Simon Riggs
suggestion. I was looking forward to the Jules Verne-like nostalgia of the other suggestion over the years to come. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Hot Standby Design

2008-09-24 Thread Simon Riggs
On Wed, 2008-09-24 at 12:35 -0400, Robert Treat wrote: On Wednesday 24 September 2008 03:27:44 Simon Riggs wrote: On Wed, 2008-09-24 at 00:30 -0400, Robert Treat wrote: * However, some WAL redo actions will be for DDL actions. These DDL actions are repeating actions that have already

Re: [PATCHES] [HACKERS] Subtransaction commits and Hot Standby

2008-09-24 Thread Simon Riggs
On Wed, 2008-09-24 at 13:48 +0100, Simon Riggs wrote: On Tue, 2008-09-23 at 22:47 +0100, Simon Riggs wrote: I've tested this some more and am much happier with it now. The concept is fine, but I've found a coding bug in further testing. Please wait now for new version before review. OK

Re: [HACKERS] Transaction Snapshots and Hot Standby

2008-09-25 Thread Simon Riggs
On Wed, 2008-09-24 at 21:19 -0400, Bruce Momjian wrote: Simon Riggs wrote: 2. Master ignores Standby's OldestXmin Effects: * Long running queries on standby... Have no effect on primary Can delay apply of WAL records on standby * Queries on standby give consistent answers

Re: [HACKERS] Add default_val to pg_settings

2008-09-25 Thread Simon Riggs
not be applied, but a similar one could and should be. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Transaction Snapshots and Hot Standby

2008-09-25 Thread Simon Riggs
On Thu, 2008-09-25 at 11:14 +0200, Zeugswetter Andreas OSB sIT wrote: Simon Riggs wrote: 2. Master ignores Standby's OldestXmin Effects: * Long running queries on standby... Have no effect on primary Can delay apply of WAL records on standby * Queries on standby give

Re: [HACKERS] Transaction Snapshots and Hot Standby

2008-09-25 Thread Simon Riggs
On Wed, 2008-09-24 at 21:22 -0400, Bruce Momjian wrote: Bruce Momjian wrote: Simon Riggs wrote: 2. Master ignores Standby's OldestXmin Effects: * Long running queries on standby... Have no effect on primary Can delay apply of WAL records on standby * Queries on standby

Re: [HACKERS] Transaction Snapshots and Hot Standby

2008-09-25 Thread Simon Riggs
of file that didn't exist when they started, so long queries need not be cancelled when they access growing tables. That combines all the suggested approaches into one. It still leaves the possibility of passing the standby's OldestXmin to the primary, but does not rely upon it. -- Simon Riggs

Re: [HACKERS] Add default_val to pg_settings

2008-09-25 Thread Simon Riggs
, no? Yes, those names seem very appropriate to me. Finding the reset_val isn't that tough, IIRC the way the guc assignment works with its doit flag. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Add default_val to pg_settings

2008-09-25 Thread Simon Riggs
On Thu, 2008-09-25 at 14:52 +0200, Magnus Hagander wrote: Simon Riggs wrote: On Thu, 2008-09-25 at 14:42 +0200, Magnus Hagander wrote: Having one column named reset_val and one named boot_val should work, no? Yes, those names seem very appropriate to me. Finding the reset_val

Re: [HACKERS] Add default_val to pg_settings

2008-09-25 Thread Simon Riggs
, but the previous discussion was all about what to call the columns ... Sorry Tom, I meant I was agreeing with both of you, not just Magnus. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] get_relation_stats_hook()

2008-09-25 Thread Simon Riggs
On Mon, 2008-09-22 at 18:41 +0100, Gregory Stark wrote: The easiest way to fix this seems like also the best way, instead of storing a boolean store the pointer to the release function. Implemented as suggested. v5 enclosed. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training

Re: [HACKERS] Missing results from scroll cursor in PostgreSQL 8.3.3?

2008-09-25 Thread Simon Riggs
a backwards scan if needed. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-26 Thread Simon Riggs
On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Version 7 Anyway, that's sufficiently bad that I'm bouncing the patch for reconsideration. No problem, I understand this needs discussion. There's less detail here than first appears

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-26 Thread Simon Riggs
On Fri, 2008-09-26 at 11:20 +0100, Simon Riggs wrote: After reading this for awhile, I realized that there is a rather fundamental problem with it: it switches into consistent recovery mode as soon as it's read WAL beyond ControlFile-minRecoveryPoint. In a crash recovery situation

Re: [HACKERS] lock contention on parallel COPY ?

2008-09-26 Thread Simon Riggs
of the index. We need to load data a block at a time and buffer the inserts into the index also, so we don't need to lock/unlock per row. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] lock contention on parallel COPY ?

2008-09-26 Thread Simon Riggs
On Fri, 2008-09-26 at 14:00 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Stefan Kaltenbrunner [EMAIL PROTECTED] writes: samples %symbol name 5552616.5614 LWLockAcquire 29721 8.8647 DoCopy 26581 7.9281 CopyReadLine 25105 7.4879

Re: [HACKERS] lock contention on parallel COPY ?

2008-09-26 Thread Simon Riggs
On Fri, 2008-09-26 at 20:07 +0200, Stefan Kaltenbrunner wrote: Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Stefan Kaltenbrunner [EMAIL PROTECTED] writes: samples %symbol name 5552616.5614 LWLockAcquire 29721 8.8647 DoCopy 26581 7.9281 CopyReadLine

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-28 Thread Simon Riggs
reached minRecoveryLoc then we continue until we find it. There is a loophole, as described on separate post, but that can be plugged by offering explicit setting of the minRecoveryLoc from recovery.conf. Most people use pg_start_backup() so do not experience the need for that. -- Simon Riggs

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Sun, 2008-09-28 at 21:16 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: It does nothing AFAICS for the problem that when restarting archive recovery from a restartpoint, it's not clear when it is safe to start letting in backends. You need to get past the highest LSN

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 08:46 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: ... That kinda works, but the problem is that restartpoints are time based, not log based. We need them to be deterministic for us to rely upon them in the above way. Right, but the performance

[HACKERS] Fatal Errors

2008-09-29 Thread Simon Riggs
then that makes some coding easier, so seems sensible to check. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] Fatal Errors

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 10:30 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Is it possible to have a FATAL error that crashes a backend and for it to *not* have written an abort WAL record for any previously active transaction? Well, a FATAL error will still go through

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: I think we can get away with writing the LSN value to disk, as you suggested, but only every so often. No need to do it after every WAL record, just consistently every so often, so it gives us a point

Re: [HACKERS] Fatal Errors

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 11:18 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Mon, 2008-09-29 at 10:30 -0400, Tom Lane wrote: Like what? For constructing snapshots during standby. I need a data structure where emulated-as-running transactions can live. If backend birth

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 11:24 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote: ... If we crash and restart, we'll have to get to the end of this file before we start letting backends in; which might be further than we actually

Re: [HACKERS] Fatal Errors

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 12:14 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: * Might we make AbortTransaction critical just as far as the END_CRIT_SECTION after XLogInsert in RecordTransactionAbort(), but no further? Don't expect yes, but seems worth recording thoughts

[HACKERS] Plans for Hot Standby

2008-10-01 Thread Simon Riggs
. Many thanks. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Block-level CRC checks

2008-10-02 Thread Simon Riggs
of performance testing. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-02 Thread Simon Riggs
On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Version 7 After reading this for awhile, I realized that there is a rather fundamental problem with it: it switches into consistent recovery mode as soon as it's read WAL beyond ControlFile

Re: [HACKERS] Transaction Snapshots and Hot Standby

2008-10-02 Thread Simon Riggs
solvable now. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Subtransaction commits and Hot Standby

2008-10-06 Thread Simon Riggs
On Sun, 2008-10-05 at 14:51 -0400, Alvaro Herrera wrote: Simon Riggs wrote: OK, spent long time testing various batching scenarios for this using a custom test harness to simulate various spreads of xids in transaction trees. All looks fine now. I had a look and was mostly rephrasing

Re: [HACKERS] Reducing some DDL Locks to ShareLock

2008-10-07 Thread Simon Riggs
On Mon, 2008-10-06 at 18:57 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: It seems possible to change some DDL commands/subcommands to use a ShareLock rather than an AccessExclusiveLock. Enclosed patch implements this reduction for CREATE TRIGGER, CREATE RULE and ALTER TABLE

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Thu, 2008-10-02 at 19:07 -0400, Andrew Dunstan wrote: Simon Riggs wrote: Just seen this patch has been bounced into November CommitFest, even though the new patch fixes all of the concerns raised. I'm concerned that this is going to make the final Hot Standby patch fairly large

Re: [HACKERS] Subtransaction commits and Hot Standby

2008-10-07 Thread Simon Riggs
On Sun, 2008-10-05 at 14:51 -0400, Alvaro Herrera wrote: Simon Riggs wrote: OK, spent long time testing various batching scenarios for this using a custom test harness to simulate various spreads of xids in transaction trees. All looks fine now. I had a look and was mostly rephrasing

Re: [HACKERS] Reducing some DDL Locks to ShareLock

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 08:30 -0400, Andrew Dunstan wrote: Simon Riggs wrote: My main focus is on these commands * CREATE TRIGGER * ALTER TABLE .. ADD PRIMARY KEY * ALTER TABLE .. ADD FOREIGN KEY because those are the most painful ones. We could make it work against more

Re: [HACKERS] Reducing some DDL Locks to ShareLock

2008-10-07 Thread Simon Riggs
, if ever, AFAICS. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 10:08 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: A patch specifically marked as required for other work has been delayed by more than 5 weeks on queue and nobody was ever assigned to review it. That was exactly the problem CommitFests were supposed

Re: [HACKERS] Reducing some DDL Locks to ShareLock

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 07:21 +0100, Simon Riggs wrote: It was an excellent question because that aspect isn't handled correctly in the enclosed patch for subcommands, other than index-creating constraints. My main focus is on these commands * CREATE TRIGGER * ALTER TABLE .. ADD PRIMARY

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 10:37 -0400, Alvaro Herrera wrote: Simon Riggs wrote: I'm just grumpy because I can't see a way to do the patch-on-patch-on-patch that I'll need to make this all work for Nov 1. So big patch here we come. But that's just the way it is and I'll stop honking about

Re: [HACKERS] Reducing some DDL Locks to ShareLock

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 10:35 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Tue, 2008-10-07 at 10:05 -0400, Robert Haas wrote: 3. The patch introduces a slight weirdness: if you create two FKs on the same column at the same time you end up with two constraints

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
to achieve. That's one reason to strip out the bgwriter stuff. It's the postmaster state change stuff that's most needed. Anyway, watch this space. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Reducing some DDL Locks to ShareLock

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 11:46 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Tue, 2008-10-07 at 10:35 -0400, Tom Lane wrote: I wonder whether this could be helped if we refactored pg_constraint. Sounds better. Doesn't make much sense as it is now. I looked at the code

Re: [HACKERS] Reducing some DDL Locks to ShareLock

2008-10-08 Thread Simon Riggs
On Tue, 2008-10-07 at 11:46 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Tue, 2008-10-07 at 10:35 -0400, Tom Lane wrote: I wonder whether this could be helped if we refactored pg_constraint. Sounds better. Doesn't make much sense as it is now. I looked at the code

[HACKERS] pg_stop_backup wait bug fix

2008-10-08 Thread Simon Riggs
Minor bug fix for pg_stop_backup() to prevent it waiting longer than necessary in certain circumstances. Was originally part of recovery_infrastructure patch. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support Index: src/backend/access/transam/xlog.c

Re: [HACKERS] Reducing some DDL Locks to ShareLock

2008-10-08 Thread Simon Riggs
On Wed, 2008-10-08 at 11:24 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: (That in itself is painful, surely DDL should fail if it sees another DDL statement in progress trying to do same thing). Surely not. The other transaction doing the DDL might roll back. Maybe so, but trying

Re: [HACKERS] [PATCHES] Infrastructure changes for recovery (v8)

2008-10-08 Thread Simon Riggs
On Wed, 2008-10-08 at 14:43 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: * optional recovery_safe_start_location parameter now provided in recovery.conf, to allow a consistency point to be manually defined if a base backup was not taken using standard pg_start/stop backup functions

Re: [HACKERS] Updates of SE-PostgreSQL 8.4devel patches (r1081)

2008-10-08 Thread Simon Riggs
a clear need for Postgres in government and hi-security businesses, so we need to get this right. But there's not much point doing 65% or 135% of what's needed. Your efforts and attention are appreciated by all. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support

[HACKERS] latestCompletedXid

2008-10-10 Thread Simon Riggs
thing. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] LWLockAcquire with priority

2008-10-10 Thread Simon Riggs
(LWLockId lockid, LWLockMode mode) LWLockAcquire() maps to LWLockAcquireWithoutPriority for all locks except for WALInsertLock and SLRU locks, which map to LWLockAcquireWithPriority(x, x, false) LockAcquireWithPriority(x, x, true) would then be used in key places as suggested above. -- Simon

Re: [HACKERS] CLUSTER, REINDEX, VACUUM in read only transaction?

2008-10-13 Thread Simon Riggs
not to bother any further. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] xact_desc

2008-10-13 Thread Simon Riggs
5702: Transaction - commit: 2008-10-14 03:14:17.687843+01; subxacts: 10447936 0 STATEMENT: commit; The arrays... work fine in recovery, just not prior to inserting. Anyway, that led me a merry dance with other code. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services

Re: [HACKERS] xact_desc

2008-10-14 Thread Simon Riggs
On Mon, 2008-10-13 at 23:05 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: ISTM that xact_desc routines do not work properly when called with WAL_DEBUG enabled from XLogInsert(). Well, now that you mention it, that code is utterly, completely broken, and always has been

[HACKERS] Deriving Recovery Snapshots

2008-10-14 Thread Simon Riggs
| 65 +++ 17 files changed, 1432 insertions(+), 44 deletions(-), 211 mods(!) Your comments are welcome, especially questions and thoughts around the correctness of the approach. Lots more comments in patch. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services

Re: [HACKERS] CLUSTER, REINDEX, VACUUM in read only transaction?

2008-10-15 Thread Simon Riggs
. So I would say ban all the utilities mentioned from read-only transactions, and don't be influenced by what non-read only transactions do. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers

[HACKERS] SlruPhysicalReadPage

2008-10-15 Thread Simon Riggs
special. I think this can cause recovery to fail *now*. What say you? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Deriving Recovery Snapshots

2008-10-15 Thread Simon Riggs
On Wed, 2008-10-15 at 12:58 -0700, Jeff Davis wrote: On Tue, 2008-10-14 at 18:50 +0100, Simon Riggs wrote: I've worked out what I think is a workable, efficient process for deriving snapshots during recovery. I will be posting a patch to show how this works tomorrow [Wed 15 Oct], just

Re: [HACKERS] SlruPhysicalReadPage

2008-10-15 Thread Simon Riggs
On Wed, 2008-10-15 at 15:13 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: This seems inconsistent. Why is the first page OK to be created, but any other pages after that cause failure? ISTM the first page is nothing special. It's special on the writing side, I'm not sure

Re: [HACKERS] Deriving Recovery Snapshots

2008-10-16 Thread Simon Riggs
On Thu, 2008-10-16 at 13:55 +0100, Simon Riggs wrote: Other related patches are * recovery_infrastruc.v9.patch * atomic_subxids.v7.patch They don't all apply cleanly together, but the changes are unrelated, so those patches can still be reviewed without wasting energy. Next phase

Re: [HACKERS] Deriving Recovery Snapshots

2008-10-16 Thread Simon Riggs
On Thu, 2008-10-16 at 15:20 +0100, Simon Riggs wrote: I've integrated my five patches together into one now: * recovery_infrastruc.v9.patch * atomic_subxids.v7.patch * hs_connect * hs_checks * hs_snapshot Seems positive that it all integrated so quickly and tests OK. More later. Wahoo

Re: [HACKERS] Deriving Recovery Snapshots

2008-10-16 Thread Simon Riggs
On Thu, 2008-10-16 at 18:52 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: Each backend that existed on the master is represented by a PROC structure in the ProcArray. These are known as recovery procs and are similar to the dummy procs used for prepared transactions. All recovery

[HACKERS] Hot Standby: First integrated patch

2008-10-17 Thread Simon Riggs
out designs for most of these aspects and will discuss them on -hackers, though most design notes are in the Wiki. I'm still looking into prepared transactions. Comments appreciated. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- -- Hot Standby tests

Re: [HACKERS] Block-level CRC checks

2008-10-17 Thread Simon Riggs
whenever you want to set CRC checks. That way you get to choose. CHECK TABLE? -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Block-level CRC checks

2008-10-17 Thread Simon Riggs
every time, with full impact. Other values delay the use of CRC checks. Kind of like freezing parameters. Let people choose. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Hot Standby: First integrated patch

2008-10-18 Thread Simon Riggs
On Fri, 2008-10-17 at 16:47 -0400, Merlin Moncure wrote: On Fri, Oct 17, 2008 at 10:38 AM, Simon Riggs [EMAIL PROTECTED] wrote: First integrated patch for Hot Standby, allowing queries to be executed while in recovery mode. The patch tests successfully with the enclosed files

Re: [HACKERS] Reducing some DDL Locks to ShareLock

2008-10-18 Thread Simon Riggs
On Tue, 2008-10-07 at 10:35 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Tue, 2008-10-07 at 10:05 -0400, Robert Haas wrote: 3. The patch introduces a slight weirdness: if you create two FKs on the same column at the same time you end up with two constraints

Re: [HACKERS] Window Functions: buffering strategy

2008-10-20 Thread Simon Riggs
of those done (row?), than to attempt all 3 and end up with none because of emerging details. Good luck. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

[HACKERS] Block level concurrency during recovery

2008-10-20 Thread Simon Riggs
to differentiate between WAL record types so we can tell which to acquire CleanupLock for and which not. * GIST doesn't use CleaupLocks at all. So I'm very unclear here. Teodor has mentioned that it should be OK for GIST/GIN. Can I double check that based upon my inspection of the code? -- Simon

[HACKERS] Index use during Hot Standby

2008-10-20 Thread Simon Riggs
() hashinsert.c _hash_doinsert() -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

<    7   8   9   10   11   12   13   14   15   16   >