Re: [HACKERS] error: could not find pg_class tuple for index 2662

2011-08-15 Thread daveg
[adding back hackers so the thread shows the resolution] On Sun, Aug 14, 2011 at 07:02:55PM -0400, Tom Lane wrote: Sounds good. Based on my own testing so far, I think that patch will probably make things measurably better for you, though it won't resolve every corner case. The most recent

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Simon Riggs
On Mon, Aug 15, 2011 at 2:31 AM, Joachim Wieland j...@mcknight.de wrote: In short, this is how it works: SELECT pg_export_snapshot();  pg_export_snapshot  03A1-1 (1 row) (and then in a different session) BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Heikki Linnakangas
On 15.08.2011 04:31, Joachim Wieland wrote: The one thing that it does not implement is leaving the transaction in an aborted state if the BEGIN TRANSACTION command failed for an invalid snapshot identifier. So what if the snapshot is invalid, the SNAPSHOT clause silently ignored? That sounds

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Heikki Linnakangas
On 15.08.2011 10:40, Simon Riggs wrote: On Mon, Aug 15, 2011 at 2:31 AM, Joachim Wielandj...@mcknight.de wrote: In short, this is how it works: SELECT pg_export_snapshot(); pg_export_snapshot 03A1-1 (1 row) (and then in a different session) BEGIN TRANSACTION

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Andres Freund
On Monday, August 15, 2011 08:40:34 Simon Riggs wrote: On Mon, Aug 15, 2011 at 2:31 AM, Joachim Wieland j...@mcknight.de wrote: In short, this is how it works: SELECT pg_export_snapshot(); pg_export_snapshot 03A1-1 (1 row) (and then in a different

Re: [HACKERS] Online base backup from the hot-standby

2011-08-15 Thread Jun Ishiduka
* Not correspond yet * full_page_write = off - If the primary is full_page_write = off, archive recovery may not act normally. Therefore the standby may need to check whether full_page_write = off to WAL. Isn't having a standby make the full_page_write = on in

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread PostgreSQL - Hans-Jürgen Schönig
On Aug 15, 2011, at 9:40 AM, Simon Riggs wrote: On Mon, Aug 15, 2011 at 2:31 AM, Joachim Wieland j...@mcknight.de wrote: In short, this is how it works: SELECT pg_export_snapshot(); pg_export_snapshot 03A1-1 (1 row) (and then in a different session)

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Florian Weimer
* Simon Riggs: I don't see the need to change the BEGIN command, which is SQL Standard. We don't normally do that. Some language bindings treat BEGIN specially, so it might be difficult to use this feature. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH

[HACKERS] pg_stat_replication vs StandbyReplyMessage

2011-08-15 Thread Magnus Hagander
The pg_stat_replication view exposes all the fields in StandbyReplyMessage *except* for the timestamp when the message was generated. On an active system this is not all that interesting, but on a mostly idle system that allows the monitoring to react faster than the timeout that actually kicks

[HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Magnus Hagander
I'm trying to make my streaming log receiver work properly with 9.1, and have come across a couple of things. The first one that's causing trouble is that the definition of the protocol is currently in walprotocol.h, which is not include:able in a frontend application. AFAICT, this is because it

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 3:47 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.08.2011 04:31, Joachim Wieland wrote: The one thing that it does not implement is leaving the transaction in an aborted state if the BEGIN TRANSACTION command failed for an invalid snapshot

Re: [HACKERS] pg_stat_replication vs StandbyReplyMessage

2011-08-15 Thread Robert Haas
On Mon, Aug 15, 2011 at 7:03 AM, Magnus Hagander mag...@hagander.net wrote: The pg_stat_replication view exposes all the fields in StandbyReplyMessage *except* for the timestamp when the message was generated. On an active system this is not all that interesting, but on a mostly idle system

Re: [HACKERS] Online base backup from the hot-standby

2011-08-15 Thread Robert Haas
2011/8/15 Jun Ishiduka ishizuka@po.ntts.co.jp: * Not correspond yet  * full_page_write = off    - If the primary is full_page_write = off, archive recovery may not act       normally. Therefore the standby may need to check whether full_page_write       = off to WAL. Isn't

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 6:41 AM, Florian Weimer fwei...@bfk.de wrote: * Simon Riggs: I don't see the need to change the BEGIN command, which is SQL Standard. We don't normally do that. Some language bindings treat BEGIN specially, so it might be difficult to use this feature. It's true,

Re: [HACKERS] SSI error messages

2011-08-15 Thread Peter Eisentraut
On lör, 2011-07-16 at 21:55 +0300, Heikki Linnakangas wrote: I think I would prefer something like this: ERROR: could not serialize access due to read/write dependencies among transactions DETAIL: Reason code: %s HINT: The transaction might succeed if retried. I've done it this way

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Robert Haas
On Mon, Aug 15, 2011 at 3:51 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: It would be nice a symmetry, but you'd need a limitation that pg_import_snapshot() must be the first thing you do in the session. And it might be hard to enforce that, as once you get control into the

Re: [HACKERS] VACUUM FULL versus TOAST

2011-08-15 Thread Robert Haas
On Sun, Aug 14, 2011 at 7:43 PM, Greg Stark st...@mit.edu wrote: On Sun, Aug 14, 2011 at 5:15 PM, Tom Lane t...@sss.pgh.pa.us wrote: There would be some merit in your suggestion if we knew that all/most toasted columns would actually get fetched out of the catcache entry at some point.  Then

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Peter Geoghegan
On 15 August 2011 12:22, Magnus Hagander mag...@hagander.net wrote: The basic reason for this is that we're putting TimestampTz fields in the protocol. This also means that the protocol actually changes definition depending on if the server is compiled with integer or float timestamps.

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: I'm trying to make my streaming log receiver work properly with 9.1, and have come across a couple of things. The first one that's causing trouble is that the definition of the protocol is currently in walprotocol.h, which is not include:able in a

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Magnus Hagander
On Mon, Aug 15, 2011 at 16:20, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: I'm trying to make my streaming log receiver work properly with 9.1, and have come across a couple of things. The first one that's causing trouble is that the definition of the

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: On Mon, Aug 15, 2011 at 16:20, Tom Lane t...@sss.pgh.pa.us wrote: However, for a narrow fix, I could see moving the data type definition to someplace with fewer dependencies.  Perhaps split it into a separate file timestamp_type.h, or something like

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: Joachim Wieland j...@mcknight.de wrote: BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ (SNAPSHOT = '03A1-1'); I don't see the need to change the BEGIN command, which is SQL Standard. No, it's not standard. To quote from our docs at:

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Magnus Hagander
On Mon, Aug 15, 2011 at 16:53, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Mon, Aug 15, 2011 at 16:20, Tom Lane t...@sss.pgh.pa.us wrote: However, for a narrow fix, I could see moving the data type definition to someplace with fewer dependencies.  Perhaps

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Steve Singer
On 11-08-15 10:00 AM, Peter Geoghegan wrote: Without commenting on what should be done in your specific case, I wonder whether it's time to fully retire the deprecated double representation of timestamps. Is anyone actually expected to rely on their availability when 9.2 is released? This also

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Heikki Linnakangas
On 15.08.2011 18:54, Magnus Hagander wrote: On Mon, Aug 15, 2011 at 16:53, Tom Lanet...@sss.pgh.pa.us wrote: Magnus Hagandermag...@hagander.net writes: I also ran into a similar problem with some WAL macro definitions that are in xlog_internal.h. I've moved them to xlogdefs.h in the attached

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Perhaps we should change the protocol so that it explicitly says which file the streamed piece of WAL belongs to. That way a client could write it to the correct file without knowing about all those macros. Yeah, maybe.

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Magnus Hagander
On Mon, Aug 15, 2011 at 18:12, Tom Lane t...@sss.pgh.pa.us wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Perhaps we should change the protocol so that it explicitly says which file the streamed piece of WAL belongs to. That way a client could write it to the correct

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Peter Geoghegan
On 15 August 2011 16:56, Steve Singer ssin...@ca.afilias.info wrote: This would mean that anyone using the floating point timestamps today won't be able to use pg_upgrade to upgrade to whichever version we remove them from.  8.3 had float based timestamps as the default and I suspect many

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: On Mon, Aug 15, 2011 at 16:53, Tom Lane t...@sss.pgh.pa.us wrote: Don't think you should expose fsec_t, nor most of those macros.  The foo_per_bar values are just namespace clutter. Hmm, ok. I just went for what seemed like a reasonable subset. I do

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Magnus Hagander
On Mon, Aug 15, 2011 at 18:40, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Mon, Aug 15, 2011 at 16:53, Tom Lane t...@sss.pgh.pa.us wrote: Don't think you should expose fsec_t, nor most of those macros.  The foo_per_bar values are just namespace clutter.

Re: [HACKERS] our buffer replacement strategy is kind of lame

2011-08-15 Thread Kevin Grittner
Greg Smith g...@2ndquadrant.com wrote: Anyway, I think every idea thrown out here so far needs about an order of magnitude more types of benchmarking test cases before it can be evaluated at all. Right. I'm very excited about all the optimizations going in, and can't see where the ones

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Steve Singer
On 11-08-15 12:33 PM, Peter Geoghegan wrote: On 15 August 2011 16:56, Steve Singerssin...@ca.afilias.info wrote: This would mean that anyone using the floating point timestamps today won't be able to use pg_upgrade to upgrade to whichever version we remove them from. 8.3 had float based

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Tom Lane
Steve Singer ssin...@ca.afilias.info writes: On 11-08-15 12:33 PM, Peter Geoghegan wrote: On 15 August 2011 16:56, Steve Singerssin...@ca.afilias.info wrote: This would mean that anyone using the floating point timestamps today won't be able to use pg_upgrade to upgrade to whichever version

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Peter Geoghegan
On 15 August 2011 18:09, Steve Singer ssin...@ca.afilias.info wrote: Really? I find that slightly surprising, considering that a quick look at master's timestamp.c suggests that the choice to use the in64 representation over the double representation is entirely a case of compile time

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Simon Riggs
On Mon, Aug 15, 2011 at 5:15 PM, Magnus Hagander mag...@hagander.net wrote: On Mon, Aug 15, 2011 at 18:12, Tom Lane t...@sss.pgh.pa.us wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Perhaps we should change the protocol so that it explicitly says which file the streamed

[HACKERS] Should we have an optional limit on the recursion depth of recursive CTEs?

2011-08-15 Thread Peter Geoghegan
Should we have an optional, disabled-by-default limit on the recursion/iteration depth of recursive CTEs to guard against stupid queries that loop ad infinitum? I've looked at other database systems that support WITH RECURSIVE queries, and this idea crops up there. For example, Firebird, the only

Re: [HACKERS] Compressing the AFTER TRIGGER queue

2011-08-15 Thread Jim Nasby
On Aug 4, 2011, at 8:40 AM, Kevin Grittner wrote: Robert Haas robertmh...@gmail.com wrote: On Wed, Aug 3, 2011 at 6:05 PM, Jim Nasby j...@nasby.net wrote: Not sure how much this relates to this discussion, but I have often wished we had AFTER FOR EACH STATEMENT triggers that provided OLD and

Re: [HACKERS] Should we have an optional limit on the recursion depth of recursive CTEs?

2011-08-15 Thread Tom Lane
Peter Geoghegan pe...@2ndquadrant.com writes: Should we have an optional, disabled-by-default limit on the recursion/iteration depth of recursive CTEs to guard against stupid queries that loop ad infinitum? I think not ... I'd suggest that an appropriate interface would be an int GUC with a

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Magnus Hagander
On Mon, Aug 15, 2011 at 21:12, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Aug 15, 2011 at 5:15 PM, Magnus Hagander mag...@hagander.net wrote: On Mon, Aug 15, 2011 at 18:12, Tom Lane t...@sss.pgh.pa.us wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Perhaps we should

[HACKERS] Re: Should we have an optional limit on the recursion depth of recursive CTEs?

2011-08-15 Thread Greg Stark
On Mon, Aug 15, 2011 at 9:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: ... and that would be a seriously bad API.  There are not SUSET restrictions on other resources such as work_mem.  Why do we need one for this? I think a better analogy would be imposing a maximum number of rows a query can

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Dimitri Fontaine
Magnus Hagander mag...@hagander.net writes: Why not have a specific protocol message to indicate a change of filename? I don't mean a WAL message, I mean a streaming protocol message. That we could have, and it would work as long as it's always sent as the first packet in any stream. If we

Re: [HACKERS] index-only scans

2011-08-15 Thread Jim Nasby
On Aug 13, 2011, at 4:31 PM, Heikki Linnakangas wrote: The example is much more realistic if the query is a fetch of N latest rows from a table. Very common use case, and the whole relation's visibility statistics are completely wrong for that query. That is somewhat compensated by the

Re: [HACKERS] walprotocol.h vs frontends

2011-08-15 Thread Simon Riggs
On Mon, Aug 15, 2011 at 10:32 PM, Magnus Hagander mag...@hagander.net wrote: At present the WALSender only sends from one file at a time, so sending a message when we open a new file would be straightforward. Are you sure? We can receive a single message spanning multiple files... You're

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Jim Nasby
On Aug 15, 2011, at 6:23 AM, Joachim Wieland wrote: On Mon, Aug 15, 2011 at 3:47 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.08.2011 04:31, Joachim Wieland wrote: The one thing that it does not implement is leaving the transaction in an aborted state if the BEGIN

Re: [HACKERS] mosbench revisited

2011-08-15 Thread Greg Stark
On Wed, Aug 3, 2011 at 7:21 PM, Robert Haas robertmh...@gmail.com wrote:  I'm kind of interested by the result, actually, as I had feared that the spinlock protecting ProcArrayLock was going to be a bigger problem sooner. I think this depends on how many connections you have. If you try to

Re: [HACKERS] Should we have an optional limit on the recursion depth of recursive CTEs?

2011-08-15 Thread Peter Geoghegan
On 15 August 2011 21:31, Tom Lane t...@sss.pgh.pa.us wrote: I'd suggest that an appropriate interface would be an int GUC with a GucContext of PGC_SUSET, so that DBAs can impose system-wide limits. ... and that would be a seriously bad API.  There are not SUSET restrictions on other resources

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 6:09 PM, Jim Nasby j...@nasby.net wrote: I suspect that all the other cases of BEGIN failing would be syntax errors, so you would immediately know in testing that something was wrong. A missing file is definitely not a syntax error, so we can't really depend on user

Re: [HACKERS] our buffer replacement strategy is kind of lame

2011-08-15 Thread Jim Nasby
On Aug 13, 2011, at 3:40 PM, Greg Stark wrote: It does kind of seem like your numbers indicate we're missing part of the picture though. The idea with the clock sweep algorithm is that you keep approximately 1/nth of the buffers with each of the n values. If we're allowing nearly all the

Re: [HACKERS] index-only scans

2011-08-15 Thread Greg Smith
On 08/11/2011 04:06 PM, Robert Haas wrote: On my laptop, the first query executes in about 555 ms, while the second one takes about 1125 ms...I expect that you could get an even larger benefit from this type of query if you could avoid actual disk I/O, rather than just buffer cache thrashing,

Re: [HACKERS] index-only scans

2011-08-15 Thread Robert Haas
On Mon, Aug 15, 2011 at 7:37 PM, Greg Smith g...@2ndquadrant.com wrote: That's 5.4X as fast; not too shabby! Awesome! And with the large difference in response time, things appear to be working as hoped even in this situation.  If you try this on your laptop, where drive cache size and

[HACKERS] VACUUM FULL versus relcache init files

2011-08-15 Thread Tom Lane
This might be the last bug from my concurrent-vacuum-full testing --- at least, I have no remaining unexplained events from about two full days of running the tests. The ones that are left involve backends randomly failing like this: psql: FATAL: could not read block 0 in file

Re: [HACKERS] Enforcing that all WAL has been replayed after restoring from backup

2011-08-15 Thread Fujii Masao
On Thu, Aug 11, 2011 at 1:34 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Hmm, that's not possible for the 'tar' output, but would work for 'dir' output. Another similar idea would be to withhold the control file in memory until the end of backup, and append it to the

Re: [HACKERS] Possible Bug in pg_upgrade

2011-08-15 Thread Bruce Momjian
Applied. --- Bruce Momjian wrote: Dave Byrne wrote: Beginning with commit 002c105a0706bd1c1e939fe0f47ecdceeae6c52d pg_upgrade will fail if there are orphaned temp tables in the current database with the message 'old

Re: [HACKERS] mosbench revisited

2011-08-15 Thread Robert Haas
On Mon, Aug 15, 2011 at 6:22 PM, Greg Stark st...@mit.edu wrote: On Wed, Aug 3, 2011 at 7:21 PM, Robert Haas robertmh...@gmail.com wrote:  I'm kind of interested by the result, actually, as I had feared that the spinlock protecting ProcArrayLock was going to be a bigger problem sooner. I

[HACKERS] src/backend/storage/ipc/README

2011-08-15 Thread Robert Haas
$SUBJECT is wildly out-of-date. Is there any point in keeping this, given the large (and actually correct) comment block near the top of sinvaladt.c? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list

Re: [HACKERS] src/backend/storage/ipc/README

2011-08-15 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: $SUBJECT is wildly out-of-date. Is there any point in keeping this, given the large (and actually correct) comment block near the top of sinvaladt.c? Huh, I never noticed that file before. Yeah, it seems useless as it stands. I wonder however if we